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ABSTRACT 

The  aim  of  this  report  is  to  provide  an  overview  of  recent  developments  in  some  key 
virtual  reality  (VR)  technologies  and  systems.  Various  definitions  of  VR  will  be 
provided.  Current  VR  hardware  (Head  Motmted  Displays,  BOOM  devices.  Stereo 
Glasses,  Convolvotron,  Gloves,  Tracking  devices)  and  VR  software  (computer  graphic 
issues,  object  representation  and  toolkits)  will  be  discussed.  Current  innovative 
systems  wlQ  then  be  presented  which  will  include  a  discussion  of  SmartScene, 
PolyShop,  Responsive  Workbench,  Virtual  Workbench  and  the  CAVE.  Finally,  a  brief 
discussion  of  applications  and  areas  of  research  will  be  presented. 
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Virtual  Reality  Technologies  and  Systems 


Executive  Summary 

This  report  describes  some  of  the  key  areas  of  Virtual  Reality  (VR).  VR  is  an 
encompassment  of  an  environment  (visual,  audio,  haptic),  plus  the  humans  within  the 
environment,  plus  the  interfaces  between  the  two.  Therefore,  a  description  of  VR  needs 
to  include  many  aspects.  Initially,  various  definitions  of  VR  are  given.  The  importance 
of  immersion  is  highlighted.  Following  this,  a  brief  description  of  the  human  senses  is 
given. 

Common  VR  hardware  is  then  discussed.  This  includes  a  discussion  of  Head  Moxmted 
Displays,  Boom  devices  and  Stereo  glasses.  As  hands  are  a  natural  interface  for 
humans,  gloves  are  discussed  in  detail  next  The  Convolvotron,  which  is  used  to 
simulate  3D  sound,  is  also  briefly  described.  An  examination  of  several  types  of 
tracking  devices  is  then  given;  their  advantages  and  disadvantages  are  highlighted. 

Under  VR  software,  many  issues  are  discussed.  Basic  computer  graphics  terms  are 
defined.  The  objects  which  inhabit  the  virtual  environment  are  described;  the 
importance  of  these  objects  having  real  world  behavioural  characteristics  is 
highhghted.  Building  virtual  environments  is  then  discussed.  Toolkits  provide  an 
easier  manner  in  which  to  construct  virtual  worlds.  Several  examples  of  toolkits  are 
provided,  including  a  discussion  of  the  WorldToolKit  (WTK)  which  has  won  awards 
for  excellence. 

Once  the  hardware  and  software  foundations  were  set,  several  innovative  systems  are 
then  described.  SmartScene  and  PolyShop  are  two  systems  which  use  VR  technology 
for  the  construction  of  terrain  databases.  The  Responsive  Workbench  and  the  Virtual 
Workbench  are  two  systems  centred  arormd  a  working  table.  These  have  been 
primarily  used  for  medical  applications.  FinaUy,  the  CAVE  system  is  described.  This 
system  is  based  around  a  room  in  which  a  computer  generates  imagery  on  three  walls 
and  the  floor.  This  system  has  been  primarily  used  for  scientific  visuahsations.  It  has 
great  potential  for  other  applications. 

The  next  two  sections  of  this  report  are  basically  a  list  of  applications  and  research 
being  carried  out  in  VR.  Under  applications,  the  defence  applications,  in  particular,  are 
Hsted.  Under  research,  only  hardware  and  software  are  considered.  The  final  section 
brings  together  all  the  aspects  discussed  in  this  report. 

The  intention  of  this  report  is  to  familiarise  the  reader  with  the  key  areas  of  Virtual 
Reality  technology  and  current  systems.  This  will  result  in  better  informed  decisions 
about  the  potential  use  of  this  technology  being  made. 
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1.  Introduction 

The  aim  of  this  report  is  to  provide  an  overview  of  recent  developments  in  some  key 
virtual  reality  (VR)  technologies  and  systems.  This  report  will  limit  its  discussion  to 
information  on  key  VR  hardware  and  software  components,  innovative  systems, 
applications  and  areas  of  research.  The  breakdown  of  this  report  is  as  follows: 

•  definitions  of  VR; 

•  brief  description  of  human  senses; 

•  VR  hardware  (Head  Mounted  Displays,  BOOM  devices.  Stereo  Glasses, 
Convolvotron,  Gloves,  Tracking  devices); 

•  VR  software  (computer  graphic  issues,  object  representation  and  toolkits); 

•  innovative  systems  (SmartScene,  PolyShop,  Responsive  Workbench,  Virtual 
Workbench  and  CAVE); 

•  applications  (including  defence);  and 

•  areas  of  research  (software  and  hardware). 

It  is  envisaged  that  the  reader  of  this  report  will  have  some  understanding  of  VR  and 
the  associated  key  technologies. 


2.  Definitions 


Fundamentally,  Virtual  Reality  (VR)  is  concerned  with  a  complete  system  in  which  not 
only  is  the  virtual  envirorunent  created  by  a  computer  (i.e.  an  image  generator),  but 
where  the  user  within  the  environment  is  also  a  prime  consideration  [Larijam  1994], 
The  interfaces  between  the  user  and  the  virtual  environment  are  also  part  of  a  VR 
system.  Since  the  user  is  within  the  environment,  a  different  perspective  of  the 
environment  is  possible  and,  as  is  often  the  case,  a  change  in  perspective  yields 
surprising  and  new  insights  [Larijani  1994,  McDowall  1994].  This  unorthodox 
perspective  is  one  of  VR's  strongest  strengths. 

In  this  report,  a  distinction  is  made  between  virtual  environments  and  VR.  A  virtual 
envirorunent  is  the  world  which  exists  entirely  within  the  memory  of  the  computer 
[Aukstakabiis  and  Blatner  1992],  while  VR  contains  this  virtual  environment  and  the 
user  within  it,  and  aU  the  interfaces  needed  to  connect  the  two.  As  discussed  later, 
examples  of  such  interfaces  are  Head-Mounted  Displays  (HMDs)  and  gloves.  In  the 
past,  the  virtual  envirorunent  term  was  used  in  order  to  disassociate  the  science  of  VR 
from  the  fantasy  associated  with  VR  through  various  movies  and  games.  In  recent 
times,  however,  the  VR  term  has  once  again  gained  prominence.  Figme  1  illustrates  a 
complete  VR  system  by  showing  the  interactions  between  a  Head-Moimted  Display, 
gloves,  audio  equipment,  tracking  devices  and  the  computer  producing  the  virtual 
environment;  this  figure  is  an  adaptation  of  the  one  foxmd  in  [Vince  1995]. 
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Figure  1:  A  typical  VR  system  [adapted  from  Vince  1995]. 
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In  the  literature  there  are  many  diverse  definitions  of  VR  [Burdea  and  Coiffet  1994, 
Ellis  1994,  Kalawsky  1993,  Larijani  1994,  Lavroff  1992,  Pimentel  and  Teixeira  1993]. 
According  to  Burdea  and  Coiffet  [1994],  VR  is 

"a  simulation  in  which  computer  graphics  is  used  to  create  a  realistic  looking  world. 

In  addition,  the  synthesised  world  is  not  static  but  responds  to  user  inputs 
(gestures,  verbal  commands,  etc.)". 

They  also  state  that  VR  is  a  high  end  user  interface  that  involves  real  time  simulation 
and  interactions  through  multiple  sensorial  channels  (visual,  auditory,  tactile,  smell, 
taste,  etc.).  Ellis  [1994]  states  that  VR  potentially  provides  a  new  communication 
medium  for  the  human-computer  interaction.  From  these  definitions,  VR  encompasses 
issues  such  as  user  interfaces,  computer  graphics  and  fidelity,  and  human  sensorial 
aspects. 

Other  definitions  emphasise  the  direct  simulation  of  the  hmnan  senses  in  order  to 
create  the  experiences  of  another  world.  For  instance,  Lavroff  [1992]  defines  VR  as  a 
system  which 

"envelops  you  in  an  artificial  world  that  feels  real,  that  responds  to  your  every  move 
much  as  the  real  world  does". 

According  to  Kalawsky  [1993],  VR  is  a 

"synthetic  sensory  experience  that  communicates  physical  and  abstract  components 
to  a  human  operator  or  participant." 

Kalawsky  also  states  that  a  true  VR  system  is  one  where  the  human  is  immersed  in  a 
computer  simulation  that  imparts  visual,  auditory  and  force  sensations  and  where  the 
interaction  with  the  components  is  through  responses  being  sensed  appropriately  and 
coupled  into  the  environment. 

However,  according  to  Pimentel  and  Tebceira  [1993],  VR  is 

"an  immersive  experience  in  which  participants  wear  tracked,  head-mounted 
displays,  view  stereoscopic  images,  listen  to  3D  sounds  and  are  free  to  explore  and 
interact  with  a  3D  world". 

This  definition,  however,  is  a  little  restrictive  in  that  it  states  that  Head-Mounted 
Displays  must  be  part  of  a  VR  system.  As  illustrated  later,  the  CAVE  and  the 
Responsive  Workbench  are  VR  systems  which  do  not  use  these  devices.  Some  people, 
in  fact,  have  an  aversion  to  wearing  large,  uncomfortable,  heavy  devices  on  their  head 
[Poston  and  Serra  1994,  Bryson  and  Feiner  1994].  Finally,  Larijani  [1994]  states  that  by 
putting  the  user  into  an  active  and  participatory  role  in  the  world  created  by  the 
computer,  VR  is  attempting  to  erase  the  boundary  between  the  human  and  the 
computer. 
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A  flight  simulator  is  commonly  associated  with  simulation  and  virtual  envirorunents. 
A  flight  simulator  uses  a  computer  to  create  a  perspective  view  of  a  3D  virtual  world; 
the  orientation  of  the  simulated  aircraft  determines  the  view  of  the  world.  As  stated  by 
Vince  [1995],  VR  extends  this  in  two  important  areas: 

•  the  degree  of  user  immersion;  and 

•  the  degree  of  interaction  the  user  has  with  the  virtual  environment. 

One  of  the  crucial  aspect  of  VR  is  immersion  [Aukstakalnis  and  Blatner  1992,  Vince 
1995].  According  to  Burdea  and  Coiffet  [1994],  VR  is  composed  of  the  three  Ts; 
interaction,  immersion  and  imagination.  Similarly,  Lavroff  [1992]  states  that 
immersion,  navigation  and  maniptdation  are  the  major  components  of  VR.  The  ability 
to  immerse  the  user  in  a  computer  generated  experience  as  an  active  participant  rather 
than  a  passive  one  is  essential  in  a  VR  system  [Vince  1995]  Two  extremes  of  immersion 
are  possible.  One  extreme  is  a  totally  immersive  VR  system  based  on  head-mounted 
displays  or  other  immersive  display  technology,  while  the  second  extreme  is  desktop 
VR  emerging  from  animated  Computer  Aided  Design  (CAD)  [Kalawsky  1993].  The 
difference  between  the  two  is  the  degree  of  immersion.  Basically  the  higher  the 
interactivity,  update  rate,  image  complexity,  sound  quality,  and  stereoscopic  and  field- 
of-view,  the  higher  the  feeling  of  immersion  [Pimentel  and  Teixeira  1993]. 

Part  of  the  feeling  of  immersion  is  the  feeling  that  the  world  changes  as  you  change. 
Thus,  immersion  is  enhanced  by  allowing  the  user's  head  movements  to  control  Ae 
gaze  direction  of  the  synthetic  images;  this  provides  the  brain  with  motion  parallax 
information  which  complements  the  other  visual  cues.  This  requires  tracking  the  user's 
head  in  real  time;  the  user's  head  must  be  S5mchronised  with  the  computer  generated 
images  [Aukstakalnis  and  Blatner  1992,  Burdea  and  Coiffet  1994,  Vince  1995].  By 
sensing  the  position  and  orientation  of  the  user's  head  with  a  tiny  sensor  and  feeding 
the  resulting  data  into  an  image  generator,  a  computer  synthesised  view  of  a  world 
from  the  user's  point  of  view  can  be  generated  [Kalawsky  1993].  In  addition,  by 
tracking  the  user's  hands  through  the  use  of  gloves  or  other  sensors,  immersion  and 
interaction  are  further  enhanced.  Glove  manipulations  (i.e.  hand  movements)  are  an 
intuitive  interface  for  humans  [Abel  et  al  1995].  Another  advantage  of  gloves  is  that 
they  can  provide  tactile  feedback  which  allows  the  user  to  "touch"  or  "feel"  an  object 
in  the  virtual  environment.  The  sense  of  sormd  also  enhances  the  illusion  if,  as  the 
user's  head  moves,  the  soimd  of  a  stationary  object  remains  stationary.  Refer  once 
again  to  Figure  1. 

Humans  are  the  users  of  VR  and  therefore  the  immersive  feel  of  a  system  must  satisfy 
the  human's  perception  of  immersion.  As  such,  our  five  senses,  in  some  fashion,  must 
be  stimulated  [Kruger  et  al  1995].  The  obvious  sense  of  sight  can  be  readily  identified 
as  being  the  visual  representation  of  the  environment.  Here,  aspects  such  as  realism 
and  fidelity  must  be  considered.  This  area  has  received  the  greatest  attention  in  the 
simulation  community.  The  sense  of  hearing  can  be  satisfied  by  various  audio 
equipment,  while  for  the  sense  of  touch/ feeling,  this  can  be  satisfied  by  haptic  and 
force  feedback  devices.  Only  recently,  the  sense  of  smell  is  being  considered  [Lavroff 
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1992].  The  sense  of  taste  has  not  been  addressed.  A  very  brief  discussion  of  the  human 
senses  are  discussed  in  the  following  section. 

Issues  relating  to  satisfying  the  human  senses  in  a  virtual  environment  will  be 
discussed  in  the  section  on  VR  Hardware.  For  the  visual  sense,  Head-Mounted 
Displays  (HMDs),  BOOM  devices  and  Stereo  glasses  will  be  discussed.  For  interaction 
and  the  touch/ feeling  sense,  gloves  will  be  discussed.  A  device  for  simulating  sound 
in  a  VR  system  wiU  then  be  briefly  described.  Tracking  the  user's  head  and  hands  is 
crucial  for  VR  and  so,  tracking  devices  will  also  be  discussed. 

Software  aspects  are  then  discussed.  Basic  computer  graphics  aspects  as  well  as  object 
]"0pj'0sentation  and  behaviour  in  a  virtual  world  are  presented.  Commercial  and  free 
toolkits  which  enable  one  to  create  a  virtual  environment  will  be  presented.  Some  of 
the  hardware  and  software  sections  are  an  expanded  version  of  [Sestito  1996]. 

Once  this  foundation  has  been  set,  several  innovative  systems  will  be  presented  and 
which  comprise: 

•  two  systems  which  use  VR  for  the  construction  of  visual  databases; 

•  two  systems  which  use  a  workbench  as  their  basic  work  area;  and 

•  a  system  which  uses  a  large  room  as  its  basis. 

Some  applications  wiU  then  be  presented.  Finally,  current  areas  of  research  wdl  be 
discussed. 


3.  Human  Senses 


3.1  Sight 

Of  aU  the  human  senses,  the  sense  of  sight  is  the  most  dramatic.  The  eyes  provides  us 
with  the  greatest  amount  of  sensory  information  about  our  physical  smrroxmdings 
[Aukstakalnis  and  Blatner  1992].  Depth  perception  by  humans  gives  them  the  ability  to 
see  scenes  in  three  dimensions  (3D)  [Larijani  1994,  Lavroff  1992]. 

In  order  for  humans  to  perceive  depth,  either  one  or  two  cooperating  eyes  can  be  used 
[Burdea  and  Coiffet  1994].  With  one  eye,  depth  is  perceived  based  on  cues  inherent  in 
the  actual  images,  such  as  shadows,  texture,  etc.  Motion  parallax  is  also  used  as  closer 
objects  seem  to  move  further  than  distant  ones  when  the  head  is  moved. 

With  two  eyes,  stereopsis  is  the  basis  of  depth  perception.  When  looking  at  an  object, 
each  of  the  eyes  receives  a  slightly  different  image  of  the  object.  The  brain  uses  the 
horizontal  shift  in  the  image  positions  registered  by  the  two  eyes  to  measure  depth 
[Burdea  and  Coiffet  1994,  Lavroff  1992].  The  greater  die  convergence,  the  closer  the 
object.  Kalawsky  [1993]  provides  the  following  definition  of  Stereopsis: 


"The  horizontal  separation  of  the  eyes  causes  objects  at  different  depths  to  fall  on 
non-corresponding  parts  of  each  retina  with  the  magnitude  of  separation  a 
junction  of  the  difference  in  depth  of  the  objects.  When  2D  scenes  containing 
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disparity  are  presented  to  the  eyes,  they  fuse  to  form  a  striking  impression  of  depth 
or  volume.  Stereopsis  does  not  provide  a  cue  to  absolute  distance  unless  other  cues 
are  present". 

The  total  viewing  angle  of  a  human  is  called  the  field  of  view.  This  is  approximately 
180  degrees  horizontally  and  150  degrees  vertically  [Burdea  and  Coiffet  1994].  Depth 
perception,  combined  with  a  wide  field-of-view  and  stereo  imaging  enhances  the 
immersive  feel  within  a  VR  system. 


3.2  Touch 

The  sense  of  touch  or  feeling  is  important  because  it  confirms  what  our  eyes  see.  In 
fact,  our  sense  of  touch  is  more  rehable  than  vision.  Humans  have  two  systems  which 
provides  us  with  haptic  information  [Aukstakalnis  and  Blatner  1992]; 

•  mechanoreceptors  -  extremely  sensitive  mechanisms  which  measure  the  pressure  or 
deformation  of  the  skin;  and 

•  proprioception  -  the  complex  sets  of  muscles  within  our  hands,  which  provide 
feedback  from  muscles  and  tendons. 

Together,  mechanoreceptors  and  proprioception  make  up  the  haptic  systems, 
providing  haptic  cues  which  convey  information  about  our  environment  [Aukstakalnis 
and  Blatner  1992]. 

Mechanoreceptors  cues  give  information  about  shape,  texture,  temperature  and  tactile 
acuity  (meastue  of  tactile  precision  -  noting  how  close  together  2  separate  points  of 
tactile  simulation  can  be  before  they  feel  hke  only  one  point).  Hundreds  of  thousands 
of  these  are  required.  Proprioception  feedback  gives  information  about  shape,  force 
and  fimmess.  Our  fingers  not  only  move  around  an  object,  but  send  information  about 
the  object  to  the  brain.  So,  the  ability  to  reach  out  and  touch  something  is  not  enough 
for  humans,  because  in  the  real  world,  when  you  reach  out  and  touch  something,  it 
touches  you  back.  Without  proprioception,  there  is  no  way  to  detect  the  botmdary  on 
an  object  in  VR  [Aukstakalnis  and  Blatner  1992]. 

Tactile  (sense  of  touch  or  pressure  applied  to  skin)  and  force  feedback  (forces  acting  on 
muscles,  tendons  and  joints)  are  both  part  of  the  haptic  perceptions.  So,  for  example, 
when  one  picks  up  a  pen,  the  sense  of  touch  communicates  the  smoothness  and  texture 
of  the  pen,  while  force  feedback  describes  the  pen's  weight  and  firmness  [Pimentel  and 
Teixeira  1993].  Therefore,  in  order  to  create  the  sensation  of  touch,  one  needs  to  have 
both  tactile  and  force  feedback  devices;  collectively  these  are  known  as  haptic  devices. 
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3.3  Sound 

Humans  use  the  sound  generated  in  their  environment  for  many  things.  Some 
examples  are  given  below  [Barfield  et  al  1995]: 

•  determining  the  size  of  a  room  dirough  the  reflection  of  the  soimd  off  the  walls  and 
ceihng; 

•  obtained  an  indication  of  the  material  of  a  room  through  the  reflection  of  soimd; 

•  the  Doppler  effect  helping  us  to  estimate  the  speed  or  closeness  of  an  approaching 
vehicle;  and 

•  the  sound  generated  by  two  objects  interacting  give  us  an  indication  of  what  the 
two  objects  are. 

Soimd,  besides  giving  humans  information  about  their  environment,  is  used  by 
humans  to  direct  their  eyes  to  the  source  of  the  soxmd.  In  a  sense,  soimd  gives  humans 
a  sense  of  direction  [Barfield  et  al  1995]. 

The  basis  of  the  localization  of  sound  by  a  human  is  performed  by  several  cues. 
Basically, 

•  the  azimuth  of  the  sound  is  determined  by  the  interaural  time  difference  (i.e.  the 
differential  delay  between  the  arrival  of  a  sound  at  the  two  ears)  and  the  interaural 
intensity  difference  (i.e.  the  differential  of  the  intensity  of  a  sound  at  the  two  ears) 
and 

•  the  elevation  of  a  sound  is  determined  by  spectral  cues  (which  result  from  the 
location-dependent  filtering  properties  of  the  head  and  pinnae). 

The  sense  of  sound  complements  our  visual  skiUs  [Barfield  et  al  1995,  Larijani  1994]. 
Sound  is  important  to  a  person's  spatial  awareness  and  is  crucial  when  visual  clues  are 
minimal  or  absent.  The  purpose  of  sound  is  to  enhance  the  illusion  of  reality  or 
augment  information  provided  to  the  user  through  other  channels.  For  realism,  the 
direction  of  the  sound  must  remain  constant,  even  though  the  head  is  moving  in  the 
environment.  This  type  of  stereoscopic  'surround-sound'  is  sometimes  called  spatial 
sound. 


3.4  Other  Senses 

The  other  two  senses  have  not  been  given  much  consideration  in  the  VR  field.  The 
sense  of  smeU  allows  humans  to  detect  odour  through  their  noses,  while  the  sense  of 
taste  is  experienced  through  the  taste  buds  on  their  tongues.  However,  some  have 
started  to  consider  methods  for  the  incorporation  of  smells  into  their  VR  systems 
[Lavroff  1992,  Moshell  et  al  1995]. 
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4.  VR  Hardware 


Lavroff  [1992]  states  that  immersion  in  VR  is  primarily  a  function  of  hardware.  Some 
systems  use  a  Head  Moimted  Display  (HMD)  to  simulate  the  visual  and  auditory 
sense  and  to  track  the  head's  movement  in  3D.  Additional  input  devices,  such  as 
gloves,  can  sense  the  hand's  positions  and  orientation.  Data  about  the  position  and 
orientation  in  three  dimensions  (3D)  of  the  user's  head  and  hands  need  to  be 
continually  fed  into  the  computer  so  that  the  visual,  auditory  and  haptic  sensations  are 
all  S5mchronised  with  the  head  and  hands  movements.  Tracking  devices  are  used  for 
this  purpose. 

The  realism  of  the  situation,  in  the  sense  of  satisfying  the  human  senses,  is  critical  for 
the  successful  'feeling'  of  immersion.  Barfield  et  al  [1995]  state  that  in  order  to  develop 
VR  hardware  and  applications,  it  is  important  to  have  a  good  xmderstanding  of  the 
human's  sensory  system  capabilities;  the  authors  provide  comparative  tables  of  current 
(1995)  VR  hardware  and  the  human  sensory  capabilities.  The  importance  of  the 
ultimate  use  of  the  systems  is  also  emphasised  by  Barfield  et  al  [1995].  They  indicate 
that  the  application  should  be  the  prime  consideration  when  considering  VR 
equipment.  For  instance,  for  a  navigation  application,  a  wide  field-of-view  is  required, 
while  for  a  tele-operated  application,  only  a  small  field-of-view  is  required;  this 
consideration  will  have  a  large  impact  upon  the  choice  of  hardware. 

According  to  Ellis  [1994],  the  illusion  of  VR  is  created  through  the  operation  of  three 
types  of  hardware: 


•  sensors  which  detect  the  operator's  body  position  and  movements; 

•  effectors  which  stimulate  the  operator's  senses;  and 

•  special-purpose  hardware  that  links  sensors  and  effectors  together  to  produce 
sensory  experiences  resembling  those  in  a  physical  environment. 

The  next  sections  describes  typical  hardware  needed  to  simulate  a  feeling  of 
immersion.  For  the  visual  sense.  Head  Mounted  Displays,  Boom  devices  and  Stereo 
glasses  are  aU  described.  Gloves  are  the  main  equipment  used  for  satisfying  the  feeling 
sense  and  for  interacting  within  the  virtual  world.  The  Convolvotron  is  briefly 
described  as  the  major  equipment  for  simulating  sotmd  in  a  3D  virtual  world.  Under 
the  tracking  sections,  six  types  of  trackers  are  discussed.  Some  final  comments  are  then 
made.  Note  that  a  good  source  of  information  about  the  companies  which  provide  VR 
hardware  is  the  VR  Sourcebook  [1996]. 
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4.1  Hardware  for  Visual  Immersion 

4.1.1  Head  Mounted  Displays  (HMDs) 

According  to  [Larijani  1994],  as  most  of  human's  ser^ory  channels  are  located  in  the 
head  (sight,  sound,  smell  and  taste),  it  follows  that  some  sort  of  headpiece  is  an 
important  part  for  an  interaction  with  the  virtual  world.  A  Head  Mounted  Display 
(HMD)  is  an  attempt  to  satisfy  these  sensory  aspects.  Basically,  the  visual  component 
of  a  HMD  is  composed  of  an  image  source  and  optics  which  relay  images  to  display 
screens. 

The  optics  of  the  HMD  [Pimentel  and  Teixeira  1993]: 

•  allow  one  to  focus  on  a  display  screen  2  to  3  inches  from  the  face;  and 

•  increase  the  field-of-view  of  the  displayed  image. 

These  special  optics  are  needed  to  allow  the  eyes  to  focus  at  such  short  distances 
without  tiring  and  are  also  needed  to  magnify  the  screen  image  to  fill  as  much  as 
possible  of  the  eye's  field-of-view.  The  optics  are  actually  placing  the  image  at  a 
comfortable  distance  for  the  eyes.  An  example  of  such  optics  are  the  LEEP  optics  [VR 
Resources  1995]  which  are  extremely  wide  output  lenses  used  to  accommodate  all  user 
intrapupiUary  distances  [Burdea  and  Coiffet  1994].  The  images  generated  by  the 
computer  are  sent  to  the  pair  of  miniaturised  displays  on  the  HMD.  As  each  eye  gets 
its  own  slightly  different  image,  the  effect  is  one  of  viewing  a  3D  world  [Larijani  1994]. 

The  computer  generates  the  images  appropriate  to  the  user's  attitude  and  position.  The 
images  are  synchronised  with  the  user's  head  movements.  The  HMD  monitors  not 
only  the  head's  orientation,  but  its  absolute  location  as  well  [Lavroff  1992].  One 
method  used  for  tracking  is  through  the  use  of  an  ultrasonic  transmitter  mounted  on 
the  HMD  and  a  receiver  moxmted  on  or  near  the  computer.  By  moving  around,  the 
computer  receives  data  which  it  uses  to  modify  the  size  of  the  objects  in  the  user's 
field-of-view.  Another  approach  for  tracking  uses  magnetic  sensors  and  magnetic  field 
surrounding  the  HMD.  As  the  user  moves  within  the  magnetic  field,  the  sensors  track 
the  user's  position  and  sends  this  information  to  the  computer  [Lavroff  1992]. 

There  are  two  types  of  HMDs.  These  are  distinguished  by  whether  the  primary  source 
of  the  images  is  either  on  or  off  the  HMD.  The  HMDs  with  the  image  source  on  the 
HMD  are  cathode-ray-tube  devices  (CRTs)  and  liquid-crystal  displays  devices  (LCD). 
Recent  developments  have  resulted  in  the  CRT  devices  being  the  preferred  option  due 
to  high  resolution  capability  and  subsequent  bright,  high  resolution  pictures  [Burdea 
and  Coiffet  1994].  However,  as  the  resolution  gap  narrows,  LCD  may  once  again  be  in 
fashion.  LCD  are  good  because  they  are  light  and  flat,  even  though  early  versions  were 
grainy  [Larijani  1994].  The  disadvantages  of  these  HMDs  is  that: 

•  they  are  usueilly  heavy;  and 

•  the  physical  danger  of  having  the  electronics  so  close  to  the  hrunan  body. 
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Figures  2  and  3  show  various  types  of  CRT  and  LCD  HMDs,  respectively;  these  figures 
were  adapted  from  Burdea  and  Coiffet  [1994]. 

The  second  type  of  HMD  where  the  primary  source  of  the  images  is  off  the  HMD  are 
fibre  optics  devices  [Burdea  and  Coiffet  1994,  Larijani  1994,  Lavroff  1992].  Here,  the 
image  is  produced  externally  and  hnked  to  the  HMD  via  fibre  optics.  The 
disadvantage  of  this  type  is  that  the  HMD  has  large  cables  protruding  from  the  HMD. 
Figure  4  shows  one  such  HMD. 

For  the  military,  the  technical  specification  for  a  HMD  is  of  prime  importance, 
followed  by  the  consideration  of  cost.  Here,  one  consideration  is  that  a  pilot  is  imable 
to  wear  any  head  piece  which  could  have  an  averse  effect  during  high  g-force 
manoeuvres.  In  addition,  the  pilot  requires  high-resolution  information  to  be 
displayed  upon  an  optically  accurate  and  rugged  display  [Vince  1995].  To  this  end,  the 
developers  of  these  systems  have  explored  the  use  of  cockpit-mounted  displays 
coupled  to  the  pilot's  helmet  through  flexible  coherent  fibre  optic  bundles  [Vince  1995]. 
The  computer  generated  images  are  then  superimposed  over  the  pilot's  view  of  the 
real  world  using  off-axis  optics  [Vince  1995].  Other  technical  specifications  include  a 
field-of-view  of  60  degrees  horizontal  and  55  degree  vertical,  with  45  degree  binocular 
overlap,  minimum  aberrations,  peripheral  vision,  eye  relief  and  scene  collimation 
[Vince  1995],  As  expected,  a  HMD  satisfying  all  of  these  requirements  would  be  fairly 
expensive  [Vince  1995]. 

The  Air  Operations  Simulation  Centre  (AOSC)  has  been  set  up  at  the  Defence  Science 
and  Technology  Organisation  (DSTO)  in  Melbourne,  Australia  [Feik  and  Mason  1993a, 
Feik  and  Mason  1993b].  The  Centre  has  purchased  a  high-end  colour  SIM-EYE™  HMD 
which  was  developed  by  Kaiser-Optics;  see  Figure  5.  This  HMD  is  a  binocular  colour 
display  system  with  miniature  cathode  ray  tubes,  collimating  optical  relays,  a  head 
tracker  and  an  audio  headset  in  a  lightweight  helmet.  To  allow  direct  viewing  of  a 
simulator  cockpit,  the  relay  optics  are  partially  transparent.  The  field-of-view  of  this 
HMD  is  60  degrees  circular  per  eye  with  a  resolution  of  1024  lines  x  1280  pixels  per 
eye.  The  colour  SIM-EYE™  HMD  satisfies  the  technical  specifications  required  for 
military  purposes  [Vince  1995].  The  HMD  will  be  used  to  present  computer  generated 
imagery  to  pilots  over  the  full  field  of  regard  available  in  real  military  aircraft. 
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CRT-biised  HMD’s:  h)  the  ”HMn-I3r'  from  Vjrhsal  ReaJit.v  Inc 
b)  the  ^'Datuvision  9c"  from  n-Visiora  Inc» 


Examples  of  CRT  Head-Mounted  Displays  [adapted  from  Burdea  and 

Coiffet  1994], 
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Fibre  0|>tics  Head  VI»unted  Display 


Figure  4:  A  fibre  optics  Head-Mounted  Display  (adapted  from  Aviation  Week  & 
Space  Technology,  Nov.  27*,  1989).  * 
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Figure  5:  The  Air  Operation  Simulation  Centre's  Head-Mounted  Display 
(SIM-EYEtm  by  Kaiser-Optics). 

Visual  immersion  relies  on  the  field-of-view,  frame  refresh  rate  and  head  tracking. 
Many  HMDs  provide  wide  horizontal  and  vertical  fields  of  view  with  a  consistent 
image  resolution.  However,  according  to  [Aukstakalnis  and  Blatner  1992],  by 
providing  a  high-resolution  inset  whenever  the  user  is  looking  and  generating  a  low- 
resolution  background  image  everywhere  else,  the  computational  load  of  the  system 
would  be  reduced.  The  field-of-view  for  high-performance  HMDs  does  not  exceed  70 
degrees  per  eye  [Larijani  1994],  The  ultimate  goal  of  these  HMD  systems  is  to  improve 
optical  systems  so  that  a  user  sees  a  180-degree  field-of-view  displayed  at  any  one 
time.  Also,  efforts  are  being  made  towards  improving  the  control  of  the  generation  of 
the  images  to  both  left  and  right  displays  [Larijani  1994]. 

For  prices  and  a  comparison  of  current  HMDs,  a  good  source  of  information  is  Ian's 
VR  buying  guide  [Buying  Guide  1996].  This  guide  provides  a  comparative  table  which 
contains  resolution,  field-of-view,  type  of  display  and  price  of  several  HMDs.  As  can 
be  seen  from  this  Hst,  HMDs  vary  in  price  from  (where  h  =  degrees  horizontal  and  v  = 
degrees  vertical): 

•  $800  (46h  and  35v,  with  428x224  resolution)  to  $8,700  for  colour  LCD  HMDs; 

•  $19,900  to  $135,000  for  colour  CRT  HMDs;  and 

•  $250,000  for  CAE's  fibre  optics  HMD  (127h  and  66v,  with  1000x1000  resolution). 

The  current  SIM-EYE  ™  HMD  with  a  resolution  of  1024  x  1280  lines  and  a  field-of- 
view  of  100  degrees  horizontal  and  50  degrees  vertical,  cost  $135,000  [Buying  Guide 
1996], 
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4.1.2  Boom  Devices 

An  interesting  variation  of  the  HMD  is  the  BOOM  device.  BOOM  is  an  acronym  for 
Binocular  Omni  Orientation  Monitor  [Larijani  1994]  and  was  developed  by  Fakespace 
Inc.  The  BOOM  is  a  floor-standing  device  consisting  of  two  rings  moimted  in  such  a 
way  (at  right  angles  to  each  other)  that  a  monitor  used  to  view  virtual  environments 
remams  suspended  in  a  horizontal  plane  between  them  regardless  of  its  platform 
motion  [Larijani  1994].  A  BOOM  is  priced  at  $74,000  [Buying  Guide  1996]. 

BOOM  uses  a  coimter-weighted  boom  design  to  eliminate  the  problems  with  optic 
weight  and  display  size.  That  is,  it  uses  an  articulated,  counterbalanced  support 
structure  and  as  such,  allows  much  higher  resolution  CRT  devices  to  be  used  instead 
of  the  low  resolution  LCD  displays  [Pimentel  and  Tebceira  1993].  This  display  supports 
up  to  1,280x1024  pixels/ eye  resolution  [VR  Resources  1995].  The  BOOM  can  be  easily 
pulled  to  the  user's  face  when  the  user  wants  to  step  into  the  virtual  world  [Nomura 
1994].  It  can  also  be  operated  with  the  hands  or  hands  free.  BOOM  devices  ranges  from 
monochrome  (BOOM)  to  full  colour  (BOOM3C).  The  six  joints  in  the  supporting 
counter-balanced  arm  enable  the  user  to  move  the  BOOM  within  a  sphere  of 
approximately  two  metre  radius  [Vince  1995]. 

Sbc  shaft  encoders  meastire  the  position  and  orientation  of  the  display  device, 
providing  a  complete  six  degree  of  freedom  sensing  (i.e.  x,  y,  z,  roU,  pitch  and  yaw). 
These  encoders  also  eliminate  most  of  the  delay  between  physically  changing  one's 
view  and  seeing  it  change  in  the  display.  Smaller  versions  that  can  attach  to  an 
engineer's  desk  are  also  possible  [Pimentel  and  Tebceira  1993]. 

4.1.3  Stereo  Glasses 

Liquid  crystal  shutter  glasses  are  a  cheaper  alternative  to  HMDs  and  BOOM  devices. 
These  glasses  are  placed  over  the  eyes.  When  wearing  such  glasses,  the  user  perceives 
objects  in  3D.  The  cost  of  such  glasses  is  $985  [Buying  Guide  1996]. 

These  liquid  crystal  glasses  are  manufactured  by  the  StereoGraphics  Corporation.  They 
work  in  the  following  manner.  Each  eye  is  shut  down  alternately  30  times  a  second.  By 
synchronising  the  glasses  with  a  TV  screen  by  way  of  an  infrared  signal  box,  the 
glasses  are  alternately  charged  to  create  an  imperceptible  shading  over  one  eye  at  a 
time.  This  shutter  effect  physically  manipulates  the  eyes  into  seeing  3D.  Also,  these 
glasses  sometimes  have  an  in-built,  sbc  degree-of-freedom  head  tracker  system  [VR 
Resources  1995]. 


4.2  Gloves 

The  use  of  gloves  adds  a  sense  of  realism  to  VR  systems.  In  addition,  the  area  of 
interaction  or  freedom  of  motion  of  a  hand  (i.e.  glove)  is  a  lot  larger  than,  say,  a  mouse. 
A  mouse  is  limited  to  2D,  while  a  glove  (i.e.  hand)  is  not;  see  Figure  6.  Additional 
degrees  of  freedom  can  also  be  obtained  by  sensing  individual  finger  movements 
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[Burdea  and  Coiffet  1994].  The  other  use  of  gloves  is  for  the  feeling  of  touch.  This  will 
be  discussed  later. 

Gloves  use  electronics  to  sense  the  position  and  orientation  of  the  hand  wearing  it.  As 
the  hand  moves  aroxmd  in  3D  space,  the  gloves  sends  data  to  a  computer  in  the  form  of 
3D  coordinates  [Aukstakalnis  and  Blatner  1992,  Lavroff  1992].  The  glove  uses  a 
tracking  mechanism  tiiat  can  use  magnetic  detection  to  determine  [Larijani  1994]: 

•  the  spatial  co-ordinates  of  the  hand's  position; 

•  where  the  hand  is,  in  relation  to  the  whole  scene;  and 

•  the  hand's  orientation  (i.e.  where  it  is  in  relation  to  the  imaginary  body). 

Gloves  are  usually  one  way  transmitters  in  that  they  only  transmit  the  data  from  the 
user  to  the  computer.  However,  some  are  also  able  to  transmit  data  from  computer  to 
user.  These  latter  gloves  increase  the  realism  of  the  virtual  environment,  as  well  as 
providing  a  mechanism  for  simulating  the  sense  of  touch  [Larijani  1994].  Individual 
fingers  can  also  be  tracked  [Burdea  and  Coiffet  1994].  The  computer  uses  the  tracked 
data  to  manipulate  an  object  in  the  virtual  world.  Movement  in  the  virtual  world  is 
synchronised  with  the  hand  movements.  Some  examples  of  well-known  gloves  are: 
VPL  DataGlove  (first  on  the  market),  CyberGlove  (replacement  for  the  DataGlove), 
PowerGlove  (arcade  games)  and  Dexterous  HandMaster  [Burdea  and  Coiffet  1994]. 

To  sense  flexing  and  movement,  various  technologies  have  been  used  with  gloves. 
Most  gloves  use  fibre  optics  [Isdale  1993,  Larijani  1994,  Lavroff  1992,  Vince  1995].  This 
type  of  glove  incorporates  a  network  of  flexible  optical  fibres  attached  to  back  of  the 
glove's  fingers  [Vince  1995].  In  order  for  the  computer  to  determine  which  joints  are 
flexed  and  to  what  extent,  the  following  technique  is  used.  A  light  of  known  intensity 
is  shone  into  one  end  of  the  network  of  fibres  and  then  measured  as  it  comes  out  of  the 
other  end  [Vince  1995].  Some  fibres  are  etched  at  finger  and  knuckle  joints  which 
results  in  a  loss  of  light  when  the  fingers  are  bent.  Light  emerging  from  the  end  of  the 
etched  fibres  is  compared  against  the  light  emerging  from  xmetched  fibres.  The 
computer  uses  these  data  to  adjust  the  display  accordingly.  This  is  how  the  DataGlove 
works  [Lavroff  1992,  Vince  1995].  Another  technique  used  by  EXO's  Dexterous 
HandMaster  involves  an  intricate  exoskeleton  of  magnets  and  sensors  that  measure  the 
bending  angle  of  each  joint  in  hand.  Several  sets  of  magnets  and  sensors  are  connected 
with  a  velcro  band  [Lavroff  1992]. 

The  ability  to  control  finger  movements  changes  from  person  to  person  and  so  any 
glove  should  be  calibrated  for  each  individual  to  ensure  accxiracy  [Larijani  1994,  Vince 
1995].  The  choice  between  the  DataGlove  and  the  HandMaster  is  dependant  upon  the 
application.  The  DataGlove  emphasises  comfort  and  style,  while  the  HandMaster 
emphasises  exacting  position  wifit  a  small  tradeoff  in  convenience  (i.e.  has  protruding 
sensors).  Figure  7  and  8  show  the  DataGlove  and  HandMaster,  respectively;  these 
figures  are  adapted  from  [Pimentel  and  Teixeira  1994].  In  1992,  tiie  cost  of  the 
DataGlove  was  $8,000US,  while  for  the  HandMaster,  the  cost  was  $15,000US  [Lavroff 
1992].  The  CyberGlove,  which  has  now  replaced  the  DataGlove,  costs  $9,800 
(18  sejasors)  or  $14,500  (20  sensors)  [Buying  Guide  1996]. 
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Figure  6:  The  work  envelope  of  a  2D  and  3D  instrument 
[adapted  from  Burdea  and  Coiffet  1994]. 
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Besides  being  able  to  sense  the  position  and  orientation  of  the  hand  wearing  the  glove, 
gloves  can  also  be  used  to  simulate  touch  or  feeling  sensations.  The  key  to  virtual 
touch  is  some  sort  of  material  pressure  or  deformation  of  the  skin.  Many  types  push 
against  the  hands  or  arms  in  order  to  register  a  mechanoreceptor  or  proprioceptive  cue 
[Aukstakalnis  and  Blatner  1992]. 

Touch  and  force  feedback,  which  are  collectively  known  as  tactile  feedback,  is  limited 
to  the  user's  hand  and  wrist.  Touch  sensors  within  the  glove  provide  information  on 
the  contact  (i.e.  surface  geometry,  smoothness  of  surface,  temperature  and  texture).  For 
the  texture,  for  example,  surface  smoothness  can  be  simulated  by  a  series  of  3D  patches 
where  height  (the  z-dimension)  characterises  the  surface  roughness  [Burdea  and 
Coiffet  1994].  Force  feedback  gives  information  on  total  contact  force  such  as  weight 
[Burdea  and  Coiffet  1994].  The  factors  recognised  as  contributing  to  tactile  sensation, 
according  to  [Larijani  1994],  are  : 

•  a  feeling  of  pressure; 

•  a  sensation  of  texture;  and 

•  an  absence  or  presence  of  heat  or  cold. 

For  realism,  both  touch  and  force  feedback  are  required.  In  the  future,  temperature  and 
spillage  will  also  need  to  be  incorporated  [Burdea  and  Coiffet  1994]. 

Haptic  feedback  devices  range  from  [Buying  Guide  1996]: 

•  $9500  for  a  tactile  on  skin  only  from  Xtensory; 

•  $14,500  for  tactile  on  hand  (i.e.  fingertips  and  palm)  from  CyberTouch;  and 

•  $100,000  for  force  feedback  on  fingers  and  wrist. 
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4.3  Three-dimensional  (3D)  Sound  Generators 

By  using  the  same  software  tools  musicians  use  to  create  computer  generated  soimds,  a 
VR  modeller  has  unlimited  flexibility.  The  most  common  way  of  controlling  and 
generating  soimds  uses  the  Musical  Instrument  Digital  Interface  (M!IDI)  standard 
[MosheU  and  Dunn-Roberts  1994,  Pimentel  and  Teixeira  1993],  With  the  computational 
power  of  modem  Digital  Signal  Processing  devices,  many  transformations  can  be 
applied  to  sound  [MosheU  and  Dunn-Roberts  1994]. 

The  Convolvotron  is  a  very  high  speed  digital  signal  processing  system  capable  of 
producing  real-time  externalised  3D  sound  cues  [Burdea  and  Coiffet  1994].  Using  a 
mathematical  model  of  sound  reflections  and  digital  sound  processing  technology,  the 
Convolvotron  is  able  to  map  one  or  more  sound  sources  to  particular  locations  in  space 
[Sound  Spatialization  1994].  It  is  capable  of  presenting  four  channels  of  real-time  3D 
sound  over  headphones.  Thus,  the  Convolvotron  produces  filtered  sotmd  that  appears 
to  come  from  different  locations  in  space  relative  to  the  listener.  The  Convolvotron 
costs  about  $15,000  [Buying  Guide  1996]. 

BasicaUy,  the  location  data  from  a  tracking  device  is  used  by  the  Convolvotron  to 
display  soxmd  at  a  particular  location.  The  Convolvotron  firstly  digitises  the  analog 
sotmd  stream,  transforms  it  to  create  the  iUusion  titat  it  originated  at  a  particular 
location,  and  then  converts  the  digital  sound  stream  back  to  analog.  This  analog  data 
can  then  be  played  in  the  stereo  headphones  of,  say,  a  HMD  [Sound  Spatialization 
1994].  The  Convolvotron  operates  at  a  peak  speed  of  320  million  operations  per  second 
in  order  to  maintain  a  real-time  response. 


4.4  Tracking  Devices 

There  are  three  ways  of  seeing  or  interacting  with  a  virtual  world  -  seeing,  hearing  and 
feeling.  (As  mentioned  before,  little  has  been  done  on  satisfying  the  sense  of  smeU  or 
taste.)  All  of  these  three  mechanisms  rely  on  knowing  the  actual  position  of  the  soiurce. 
In  addition,  being  able  to  track  the  user's  head  and  hands  in  the  virtual  world  in  real 
time  is  also  important.  This  is  because  the  user's  head  and  hands  must  be  synchronised 
with  the  computer  generated  images  [Vince  1995].  So,  by  sensing  the  position  and 
orientation  of  the  user's  head  and  hands  with  a  tiny  sensor  and  feeding  the  resulting 
data  into  a  computer  graphics  system,  a  computer  synthesised  view  of  a  world  from 
the  user's  point  of  view  can  be  generated  [Kalawsky  1993]. 

Position/ orientation  tracking  requires  a  data  set  of  six  numbers  that  need  to  be 
measured  sufficiently  fast  as  an  object  moves  at  high  speed.  These  measurements  need 
to  be  non-contact  [Burdea  and  Coiffet  1994].  These  six  numbers  consist  of 
[Aukstakalnis  and  Blatner  1992,  Burdea  and  Coiffet  1994]: 

•  three  movement/ translation  functions,  i.e.  forward/backward,  left/ right  and 
up/ down  (i.e.  x,  y,  z);  and 

•  toee  turning  or  rotational  functions  (i.e.  one's  head  or  hands  can  roll,  pitch  or  yaw). 
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These  numbers  relate  to  the  six  degrees  of  freedom  that  the  user  has  in  the  virtual 
environment  [Aukstakalnis  and  Blatner  1992],  Figure  9  illustrates  these  six  dimensions. 

Trackers  are  a  physical  device  which  are  attached  to  the  object  or  user  so  that  head  or 
hands  movements  can  be  detected.  The  tracking  needs  to  be  done  in  real-time.  For  a 
smooth  simulation  one  needs  at  least  24  (i.e.  movie)  or  better  30  frames  per  seconds 
[Burdea  and  Coiffet  1994].  Vince  [1995]  recommends  an  update  rate  of  50  updates  per 
second.  The  Air  Operations  Simulation  Centre  runs  its  simulations  at  60  updates  per 
second  [Feik  and  Mason  1993a,  Feik  and  Mason  1993b]. 


Z 


Figure  9:  An  illustration  of  the  six  degrees-of-freedom. 

4.4.1  Methods  used  for  tracking 

There  are  six  basic  position-sensing  methods  in  use  :  magnetic,  ultrasonic,  mechanical, 
optical,  inertial  and  image  extraction  [Aukstakalnis  and  Blatner  1992].  A  summary  of 
the  resolution,  accuracy  and  latency  of  the  magnetic,  ultrasonic  and  mechanical  are 
Hsted  in  Table  1  (where  figures  are  taken  from  [MosheU  and  Dunn-Roberts  1994]).  The 
six  types  of  trackers  are  described  below. 

Table  1:  A  summary  of  the  characteristics  of  the  magnetic,  ultrasonic  and  mechanical  tracking 
mechanisms  [adapted  from  Moshell  and  Dunn-Roberts  1994]. 


Resolution 

Accuracy 

Latency 

magnetic 

1mm  +  0.03 

3mm  +  0.1 

50  msec 

ultrasonic 

lOmm  +  0.5 

limited  by  variation  in 
speed  of  soxmd  due  to  air 
density 

30  msec 

mechanical 

1  degree  at  each  joint 

4mm 

<  1  msec 
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Magnetic  Trackers:  The  magnetic  (i.e.  electromagnetic)  trackers  are  the  most  popular 
because  of  the  sensor's  small  size  and  freedom  of  movement.  These  trackers  use  a  low 
frequency  signal  generated  by  a  control  box,  plus  six  wire-coil  sets  (three  for 
transmitting  and  three  for  receiving)  [Pimentel  and  Teixeira  1993].  The  resulting 
electrical  current  and  magnetic  fields  can  then  be  used  to  determine  position  and 
orientation.  Thus,  it  uses  magnetic  fields  emitted  by  a  small  antenna  on  the  control  box 
and  detected  by  one  or  more  receiving  antennas.  Most  magnetic  trackers  use 
alternating  current  (AC)  magnetic  fields  for  measurements  (e.g.  Polhemus  systems); 
another  approach  uses  direct  current  (DC)  (e.g.  Ascension  systems)  [Aukstakalnis  and 
Blatner  1992].  Several  objects  can  be  tracked  using  the  same  control  box  [Pimentel  and 
Tebceira  1993].  This  tracker  does  not  rely  on  line-of-sight  to  be  able  to  track.  However, 
it  has  a  problem  with  its  sensitivity  to  large  metal  objects  or  other  magnetic  fields 
[Pimentel  and  Teixeira  1993,  MosheU  and  Dunn-Roberts  1994]. 

Ultrasonic  Trackers:  The  ultrasonic  tracker  uses  an  ultrasonic  transducer  which  emits 
a  high  frequency  sound  pulse.  This  pulse  is  picked  up  by  three  microphones  attached 
to  the  ceiling.  The  time  delay  (i.e.  distance)  can  thus  be  measured.  However,  this  type 
of  trackers  must  remain  in  the  line-of-sight  of  the  microphones  [Aukstakalnis  and 
Blatner  1992,  MosheU  and  Dunn-Roberts  1994].  Also,  this  tracker  is  susceptible  to 
external  noise  [Pimentel  and  Teixeira  1993]. 

Mechanical  Trackers:  A  direct  mechanical  connection  between  a  reference  point  and 
the  object  to  be  tracked  is  the  basis  of  these  trackers.  It  uses  a  rigid  framework  both  to 
support  the  viewing  device  and  to  measure  position  and  orientation  [MosheU  and 
Dunn-Roberts  1994].  This  tracker  suffers  from  a  Umited  range  of  motion  [Aukstakalnis 
and  Blatner  1992,  Pimentel  and  Teixeira  1993]. 

Inertial  Trackers:  Another  mechanical  approach  is  used  by  the  inertial  trackers  which 
use  miniature  gyroscopes  to  measure  yaw,  pitch  and  roU.  These  are  thus,  not  true  six 
degree-of-freedom  trackers  because  they  can  only  measure  orientation  and  not 
translation.  However,  they  can  be  effective  where  position  information  is  rarely 
required  [Pimentel  and  Teixeira  1993].  This  type  of  tracker  needs  to  be  further 
researched  in  order  to  be  used  for  the  outside  [MosheU  and  Dunn-Roberts  1994]. 

Optical  Trackers:  The  optical  trackers  are  infrared  trackers  which  determine  the 
position  of  targets  via  triangulation  technology  from  cameras  at  known  locations.  The 
orientation  is  determined  by  observing  multiple  targets.  These  trackers  use  expensive 
signal-processing  hardware.  Optical  tracking  has  been  primarily  used  in  cockpit 
simulators  where  the  range  of  head  motion  is  fairly  limited,  but  fast  update  rates  are 
important  [Pimentel  and  Teixeira  1993].  These  trackers  are  fairly  accurate 
[Aukstakalnis  and  Blatner  1992].  However,  fine  measurements  are  difficult,  without  a 
large  separation  between  the  targets  [MosheU  and  Dunn-Roberts  1994].  Also,  these 
trackers  have  only  a  smaU  working  volume  of  approximately  one  metre  cubed.  As 
weU,  these  trackers  are  susceptible  to  line-of-sight  occulsion  [MosheU  and  Dunn- 
Roberts  1994,  Pimentel  and  Teixeira  1993]. 
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Image  Extraction  Trackers:  The  most  calculation  intensive  tracker  is  the  image 
extraction  tracker.  It  is  the  easiest  to  use  from  a  user's  viewpoint,  but  the  hardest  from 
the  developers.  A  video  camera  or  a  set  of  cameras  pointed  at  the  participant  captures 
video  images  describing  where  the  person  is  and  what  the  person  is  doing.  Visual 
processing,  which  is  extremely  difficult,  is  then  used.  It  has  great  potential  due  to  its 
relative  simplicity  [Aukstakalnis  and  Blatner  1992]. 

4.4.2  General  considerations 

In  general,  many  tracking  devices  require  clear  line-of-sight  at  all  times  between  sensor 
and  user  [Larijani  1994].  General  trackers  work  reliably  up  to  2  to  3  metres  in  perfect 
environment,  but  in  practise  this  is  reduced  by  local  electromagnetic  noise.  According 
to  Vince  [1995]: 

•  translational  accuracy  is  in  order  of  2.5  mm; 

•  translational  resolution  is  in  order  of  0.8mm; 

•  angular  ranges  cover  180  degrees  for  yaw  and  roll,  with  an  accuracy  of  0.5 
degrees;  and 

•  angular  ranges  cover  90  deg  for  pitch,  with  an  accuracy  of  0.5  degrees. 

Data  from  the  tracker  are  continuously  sent  in  a  streaming  mode.  This  is  most 
appropriate  and  necessary  for  fast  moving  objects  or  when  application  requires  a  quick 
response  to  a  change  in  the  moving  object's  position.  This  requires  a  lot  of 
communication  and  can  be  a  problem.  Burdea  and  Coiffet  [1994]  suggest  an  alternative 
method  of  xxsing  polling;  here,  data  are  only  send  whenever  the  host  requires  it. 

The  speed  in  which  the  virtual  scene  changes  depends  on  the  speed  with  which  the 
sensor  measxues  motion.  This  is  dependant  upon  two  parameters  [Burdea  and  Coiffet 
1994]: 

•  sensor  update  rate  (measurements  per  second);  and 

•  the  sensor's  latency  (time  delay  between  action  and  result). 

This  speed  defines  the  interactivity  between  the  user  and  the  computer  generating  the 
simulated  virtual  world.  This  simulation  latency  is  very  important  Minimal  latency  is 
obviously  required.  Related  to  this  is  also  the  acouacy  of  the  sensor.  The  more  accurate 
the  sensor  for  determining  the  difference  between  actual  and  measured  positions,  the 
better  the  simulation  follows  the  user's  real  actions. 

Latency  can  be  defined  as  the  time  delay  between  when  a  sample  is  taken  and  when 
the  data  is  available  for  processing.  The  total  latency  of  a  sensor  is  the  sum  of  the 
effects  of  sensor  latency,  transmission  delays  (to  and  from  the  VR  engine),  plus  the 
time  it  takes  to  re-compute  and  display  a  new  frame  [Brudea  and  Coiffet  1994].  If 
latency  is  >  100  msec,  simulation  quality  degrades  significantly  and  may  result  in  user 
dizziness  and  sickness.  Low  latency  and  fast  refresh  rates  require  a  VR  engine  that  has 
fast  CPUs  (for  modelling  of  world  d5mamics  and  for  tool  input/output)  as  well  as 
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powerful  graphics  accelerators  (for  drawing  fast  images).  Frame  refresh  rate  is 
inversely  proportional  to  scene  complexity. 

Two  well-known  trackers  have  the  following  update  rates: 

•  Polhemus  FasTrak  updates  at  120  Hz;  and 

•  Ascension  Flock  of  Birds  at  144  Hz. 

Both  Polhemus  and  Ascension  claim  a  latency  of  less  than  10ms.  Thus,  for  a  30  Hz 
visual  update  rate,  this  is  acceptable  if  data  can  be  processing  and  delivered  to  the 
computer  generating  the  image  (IG)  is  less  than  10ms,  gives  a  total  of  20ms  latency 
[Vince  1995].  The  FasTrak  costs  $6000,  while  the  Flock  of  Birds  costs  about  $2700 
[Buying  Guide  1996]. 


4.5  Final  Comments 

According  to  Barfield  et  al  [1995],  current  VR  hardware  is  only  stimulating  the  visual 
and  auditory  senses.  Pimentel  and  Tebceira  [1993]  also  indicate  that  the  object's 
physical  properties  (e.g.  weight,  composition  and  contact  with  other  sources)  need  to 
be  implemented  in  a  VR  system.  In  order  to  have  realism,  therefore,  the  sense  of 
feeling  needs  to  be  stimulated.  Thus,  both  tactile  and  force  feedback  are  crucial.  Others 
have  also  emphasised  this  point  as  researchers  are  attempting  to  develop  better  haptic 
interfaces  [Boman  1995].  In  fact,  according  to  MosheU  and  Dunn-Roberts  [1994],  the 
biggest  limiting  factor  in  training  is  the  requirement  for  better  force  feedback  based  on 
re^-time  physical  models. 


5.  Software  Considerations 

According  to  Zyda  et  al  [1993],  there  are  three  phases  involved  in  the  construction  of  a 
virtual  world: 

•  hardware  phase  -  items  such  3D  input  devices  (i.e.  gloves,  trackers,  etc)  and  3D 
output  devices  (i.e.  HMDs)  are  considered; 

•  software  phase  -  the  development  of  software  to  support  the  large  scale,  networked 
and  multi-party  virtual  environment;  and 

•  final  integration  phase  where  the  software  developed  in  phase  two  is  used  for  the 
coiistruction  of  a  fully  detailed,  fully  interactive  and  seamless  virtual  environment. 

Phase  1  has  had  the  greatest  attention  with  the  development  of  various  HMDs,  gloves, 
trackers  and  etc.  Phase  2  is  the  hardest  in  that  it  provides  the  foxmdation  of  the  virtual 
world.  Further  developments  or  refinements  to  the  software  are  required  for  phase  3, 
where  the  aim  is  to  have  things  done  with  minimal  delay  or  hesitation  [Zyda  et  al 
1993]. 
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The  software  of  the  virtual  environment,  according  to  Ellis  [1994]  must  address  three 
separate  functions: 

•  shape  and  kinematics  of  entities  (i.e.  objects  and  actors)  within  world; 

•  their  interactions  among  themselves  and  with  the  environment  according  to  real 
world  rules;  and 

•  extent  and  character  of  the  enveloping  environment. 

This  section  attempts  to  highlights  the  various  software  aspects  which  need  to  be 
considered  when  developing  a  VR  system.  As  stated  above,  Zyda  et  al  [1993]  and  Ellis 
[1994]  have  highlighted  many  important  issues.  Further  ones  are  discussed  below. 
Once  this  fotmdation  has  been  set,  the  toolkits  which  are  available  for  creating  VR 
systems  will  be  discussed 


5.1  Basic  Aspects 

This  section  will  attempt  to  describe  some  of  the  basic  computer  graphics  aspects 
needed  when  creating  a  virtual  environment.  Firstly,  the  viewpoint  of  an  observer  in 
the  virtual  worlds  needs  to  be  determined.  This  is  crucial  as  it  determines  fhe 
orientation  of  objects  in  the  world. 

Besides  the  viewpoint,  the  actual  geometry  of  an  object  must  also  be  considered.  Any 
3D  structure,  whether  a  house  or  a  landscape,  can  be  described  as  a  series  of  3D 
coordinates  values.  Any  point  in  space  can  be  defined  by  any  three  points  (i.e.  x,  y  and 
z).  Simple  rules  of  perspective  guide  the  computer  in  creating  the  2D  image  on  a  view 
screen  using  3D  information  defining  the  size  and  placement  of  the  object.  This  is  a 
method  of  generating  3D  graphics. 

The  rendering  or  drawing  of  these  images  is  also  crucial.  This  involves  examining  the 
3D  model  or  geometry  information  and  determining  how  to  draw  the  3D  models  on 
the  2D  screen  of  a  display  device.  Thus,  this  is  the  process  of  calculating  the  image 
details  and  drawing  them  on  a  screen.  The  difficult  part  is  doing  the  rendering  quickly 
enough  for  VR  systems  which  require  real-time  interaction  [Pimentel  and  Teixeira 
1993]. 

Each  object  in  the  virtual  environment  is  made  up  of  one  or  more  polygons.  A  polygon 
is  a  2D  segment  inside  a  3D  world  that  helps  make  up  a  3D  object  [Aukstakalnis  and 
Blatner  1992].  For  example,  a  cube  has  six  faces;  each  face  could  be  represent  by  a 
separate  polygon.  The  polygon  are  fitted  together  to  approximate  the  objects  in  the 
virtual  world.  For  instance,  to  construct  a  hUl,  several  polygons  could  be  fitted  together 
to  approximate  the  shape  of  the  hill.  A  larger  number  of  smaller  polygons  would 
provide  a  more  natural  looking  hill.  Flowever,  using  these  smaller  polygons  would 
mean  that  the  world  would  take  longer  to  construct.  On  the  other  hand,  too  few 
polygons  would  make  the  hill  appear  unrealistic.  Therefore,  the  number  of  polygons 
used  is  a  tradeoff  between  realism  and  speed.  In  addition,  to  make  a  lull  seem  more 
natural,  each  polygon  can  be  coloured  a  different  shade  based  on  how  the  light  strikes 
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its  surface.  Here,  the  position  and  source  of  the  light  source  needs  to  be  taken  into 
consideration. 


5.2  Objects  in  the  Virtual  Environments 

There  are  two  parts  of  an  object  inhabiting  a  virtual  world.  The  shape  of  the  virtual 
objects  needs  to  be  considered  (i.e.  number  of  polygons,  triangles,  vertices,  etc)  as  well 
as  their  appearance  (i.e.  texture,  surface,  reflection,  coefficients,  colour,  etc)  [Asch 
1992]. 

The  object  shape  can  be  created  from  scratch;  this  can  be  tedious  [Burdea  and  Coiffet 
1994].  The  level  of  detail  of  the  object  shape  is  determined  by  the  number  of  polygons 
used  to  construct  the  object.  Basically,  there  are  two  sets  of  data  needed  to  define  an 
object;  one  contaming  the  vertex  locations  of  the  object  and  the  other  containing  the 
vertex  connectivity.  These  numbers  can  then  be  used  to  transform  the  object  into  a 
wireframe  (i.e.  see-through)  or  shaded  model  of  the  object.  Unfortunately,  these 
numbers  are  not  commonly  available  from  existing  modeling  programs  [Burdea  and 
Coiffet  1994].  Alternatively,  the  image  of  an  object  can  be  scanned  into  a  computer. 
This  image  can  then  be  cut  and  shaped  to  fit  a  polygon  [Pimentel  and  Teixeira  1994]. 
Another  approach  for  obtaining  realistic  objects  is  by  importing  the  objects  through 
Computer  Aided  Design  (i.e.  CAD)  programs  [Vince  1995]. 

The  key  to  a  virtual  object's  appearance  is  its  surface  reflectivity  and  texture.  The 
surface  reflectivity  needs  to  take  into  accoimt  the  position  and  type  of  light  source. 
Depending  upon  this,  the  reflection  of  light  off  the  object  can  be  determined.  The 
texture  of  an  object  can  increase  the  level  of  detail  and  realism  of  a  scene.  Texture  also 
can  provide  3D  spatial  cues.  This  substantially  reduces  the  number  of  polygons  in  the 
scene  and  as  such,  the  refresh  rate  can  therefore  be  increased  [Burdea  and  Coiffet 
1994].  The  texture  used  by  the  objects  can  be  created  in  several  ways  [Burdea  and 
Coiffet  1994]: 

•  use  a  paint  program  to  create  and  store  the  texture  as  a  bitmap; 

•  use  a  photo  of  desired  texture  and  scan  the  photo  into  the  computer;  or 

•  use  a  commercial  texture  database. 

In  order  to  simulate  the  surface  smoothness  of  an  object,  the  texture  can  also  be 
simulated  by  a  series  of  3D  patches  where  the  height  (z-dimension)  of  the  objects 
characterises  the  surface  roughness  [Burdea  and  Coiffet  1994]. 

Finally,  determining  whether  an  object  is  static  or  dynamic  should  also  be  considered 
[Kalawsky  1993].  For 
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surface  deformations,  and  etc,  and  as  such,  objects  in  the  virtual  world  need  to  emulate 
these  characteristics  [Burdea  and  Coiffet  1994]. 


5.3  Object  Behaviours 

Some  believe  that  the  objects  in  the  virtual  environment  must  be  realistic  in  order  to 
provide  a  truer  sense  of  immersion.  Therefore,  according  to  Vince  [  1995],  any  VR 
system  must  follow  the  rules  of  real  physical  simulation;  that  is,  procedmes  used  to 
animate  virtual  environments  must  have  credible  simulated  behaviours.  Another 
consideration  is  whether  an  object  is  dependant  or  in-dependant  upon  user  inputs.  For 
instance,  a  clock  is  not  reliant  upon  any  user  inputs  and  can  be  modelled  upon  external 
sensors.  Other  virtual  objects,  however,  have  a  degree  of  independence  of  user  actions; 
that  is  a  certain  degree  of  intelligence.  In  these  cases,  the  behaviour  needs  to  be 
modelled  by  programming  virtual  reactions  or  reflexes  [Burdea  and  Coiffet  1994]. 

The  objects  in  the  virtual  world  must  be  modelled  physically  [Vince  1995].  Thus, 
virtual  objects  need  to  specify  their  mass,  weight,  inertia,  surface  texture  (smooth  or 
rough),  compliance  (hard  or  soft)  and  deformation  mode  (elastic  or  plastic).  These 
features  can  then  be  merged  with  the  geometric  modeling  of  the  object  and 
behavioural  laws  to  form  a  realistic  virtual  model.  The  real  physical  constraints  of 
objects  also  need  to  be  modelled  [Vince  1995].  For  instance,  a  draw  in  a  cupboard  is 
hmited  in  movement  away  from  the  cupboard.  Thus,  an  object  in  the  virtual  world 
should  be  similarly  constrained  to  limit  translations  and  rotations  about  the  axes  of  its 
local  frame  of  reference.  This  can  be  set  as  parameters  within  the  database  and  used  to 
ensure  that  matrix  operations  transform  the  objects  within  die  desired  Limits  [Vince 
1995].  Related  to  this,  an  object's  weight  should  be  centred  on  the  object's  center  of 
gravity. 

The  issues  of  surface  deformation  and  compliance  need  to  be  elaborated  on.  To 
simulate  surface  deformation  graphically,  each  object  in  the  virtual  world  needs  to  be 
'meshed'  and  not  represented  as  primitives  [Vince  1995].  For  example,  a  ball  cannot  be 
modelled  simply  by  a  sphere  primitive.  Each  surface  has  to  have  meshed  surface  with 
the  number  of  polygons  determined  by  the  desired  level  of  detail  and  available 
graphics  accelerators.  So,  for  example,  when  squeezing  a  ball,  the  appropriate  bulging 
is  shown. 


5.4  Building  the  Virtual  Environment 

The  database  of  a  VR  system  provides  the  content  of  the  world.  It  has  two  aspects: 

•  content  -  a  collection  of  facts  and  figures;  and 

•  expertise  -  the  accumulated  imderstanding  that  has  been  gained  through  experience 
or  study ;  this  represents  the  sum  of  what  is  perceived,  discovered  or  inferred. 
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According  to  Larijani  [1994],  both  content  and  the  knowledge  base  (i.e.  the  expertise) 
are  required  for  the  development  of  effective  VR  applications.  The  designers  of  virtual 
environments  need  specific  information  for  the  virtual  objects  they  build.  The 
information  held  in  the  database  is  used  both  for  building  the  virtual  objects  and 
environments,  and  for  supplying  information  about  those  objects  to  the  users  in  the 
virtual  world.  Databases  are  increasing  complex,  and  as  such,  the  user  needs  efficient 
interfaces  and  embedded  intelligence  (inference  techniques)  in  order  to  interact  with 
them  [Larijani  1994].  Of  prime  importance  is  the  real-time  interaction  capability 

[Mahoney  1995]. 

VR  systems  assembles  pieces  of  the  world  by  calculating  how  each  polygon  would 
look  if  viewed  from  a  certain  distance  and  direction.  Next,  all  polygons  are  sorted  so 
that  the  one  furthest  from  user's  viewpoint  are  drawn  first.  The  entire  process  is 
repeated  every  time  the  viewpoint  changes  (e.g.  moving  a  HMD  in  VR).  Obviously,  the 
number  of  polygons  is  crucial  and  this  will  influence  the  update  rate  that  is  possible 
[Pimentel  and  Teixeira  1994]. 

For  realistic  models  and  virtual  environments,  a  large  number  of  polygons  are 
required;  the  more  the  number  of  polygons  the  more  realistic  the  model.  Rendering  or 
drawing  these  polygons  at  interactive  (and  therefore  real-time)  rates  is  costly  (if  not 
impossible).  In  addition,  the  large  memory  requirements  of  this  rendering  may  not  fit 
into  the  random  access  memory  allocation  of  the  computer  [Burdea  and  Coiffet  1994]. 
This  will  result  in  time-consiuning  movements  between  the  rendering  processes  and 
the  other  memory  locations  (i.e.  besides  the  random  access  memory).  Therefore  it 
becomes  necessary  for  the  developer  to  manage  the  world  model  in  a  better  manner  in 
order  to  maintain  interactivity  at  reasonable  rates.  Several  approaches  to  solve  this 
problem  have  been  proposed.  This  involves  ceU  segmentation,  variable  detail  and 
resolution  display,  offline  pre-computation  and  memory  management  techniques 
[Burdea  and  Coiffet  1994]. 

For  instance,  ceU  segmentation  is  a  technique  which  partitions  the  virtual  world  into 
smaUer  'universes'  or  'ceUs'  [Burdea  and  Coiffet  1994].  This  attempts  to  aUeviate  the 
problem  with  rendering  large,  complex  imagery.  CeU  segmentation  has  the  advantage 
that  only  the  objects  in  the  current  'universe'  need  to  be  displayed.  This  reduces  the 
complexity  of  the  world,  significantly.  Also,  some  of  tiie  object  does  not  contribute  at 
aU  to  any  given  image,  given  that  some  of  the  object  may  be  occluded  by  another  object 
and  that  part  of  the  object  is  behind  itself. 

The  aim  of  a  real-time  interactive  VR  system  is  to  have  the  system  update  rate  running 
as  high  as  possible.  Showing  only  a  certain  level  of  detail  is  another  interesting 
technique  in  which  to  reduce  the  image  generation  load.  This  strategy  stores  within  the 
database,  different  levels  of  detail  for  specific  objects.  The  different  models  are  selected 
at  appropriate  times  [Vince  1995].  Strategies  for  simulating  such  switching  are  based 
on  distances  from  an  observer's  point  of  view  [Burdea  and  Coiffet  1994].  Smular 
approaches  can  be  applied  to  moving  objects.  Here,  objects  moving  quickly  are  seen 
only  for  a  short  time  and  appear  blurred  and  as  such,  it  is  not  necessary  to  represent 
these  objects  in  great  detail  [Vince  1995].  Rapidly  moving  objects  can  thus  be 
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represented  with  just  a  few  polygons  [Burdea  and  Coiffet  1994].  However,  by  selecting 
different  levels,  a  particular  feature  may  be  lost  during  the  transition  between  objects 
of  different  levels  of  fidelity  [Vince  1995]. 

According  to  Pimentel  and  Tebceira  [1994],  few  VR  systems  have  mastered  the 
complexities  of  handling  the  real-time  interactions  and  performing  real-time  texture 
mapping.  The  whole  process  is  complicated,  computationally  intensive  and  requires 
specialised  hardware.  Thus,  few  3D  modelling  tools  understand  the  challenges  of  not 
only  building  the  virtual  worlds,  but  also  efficiently  managing  a  complex  database  of 
objects  and  attributes  [Pimentel  and  Teixeira  1994].  High  level  object  orientated 
techniques  are  ideally  suited  to  modeUing  interactive  VR  systems  [Kalawsky  1993]. 
Here,  each  object  in  the  virtual  world  will  have  its  own  set  of  attributes  and  may  be 
part  of  another  object,  grouped  into  some  form  of  hierarchy.  Parent  attributes  may 
even  be  inherited  by  the  lower  level  objects.  These  techniques  have  the  potential  of 
alleviating  some  of  the  current  problems  with  building  virtual  environments. 

Mahoney  [1995]  also  states  that  when  building  virtual  worlds,  it  is  better  to  use  a  top- 
down  approach  as  opposed  to  a  bottom-up  approach;  that  is,  one  should  consider  the 
purpose  of  the  environment  and  then  start  the  modelling.  Barfield  et  al  [1995]  also 
emphasis  this  view.  So,  for  instance,  in  some  situations  accurate  spatial  orientation  and 
location  of  objects  is  more  important  than  detail  and  resolution.  Also,  when  modelling 
real-world  situations  the  laws  of  physics,  for  instance,  need  to  be  simulated  as  these 
give  realism  to  the  simulation  [Mahoney  1995,  Durlach  et  al  1992].  Finally,  the  success 
of  a  VR  system  is  a  fxmction  of  the  richness  of  the  experience  as  a  whole,  and  not  the 
individual  models  [Mahoney  1995]. 


5.5  High  Level  Taxonomy 

Zyda  et  al  [1993]  indicates  that  there  are  six  aspects  of  the  software  phase  that  need  to 

be  considered.  These  are  briefly  discussed  below: 

•  Interaction  -  the  aim  of  these  methods  is  to  (i)  take  the  raw  data  from  the  input 
devices  and  (ii)  convert  the  data  into  a  form  which  can  be  used  by  the  system  or 
application  so  that  some  meaningful  operation  is  executed. 

•  Navigation  -  here,  the  main  concern  is  how  one  moves  tiurough  the  3D  virtual 
world.  Aspects  such  as  the  3D  input  devices,  the  viewpoint  of  &e  virtual  camera 
and  methods  for  minimising  polygon  flow  need  to  be  considered.  All  of  these 
aspects  need  to  be  coupled  and  achieved  in  real-time. 

•  Communication  -  passing  changes  in  the  world  model  to  other  players  is  the  main 
concern  here.  Aspects  such  as  database  consistency  and  communication  protocols 
needs  to  be  considered. 

•  Autonomy  -  for  many  applications  it  is  useful  to  have  autonomous  agents 
interacting  with  the  real  live  agents.  The  issues  of  how  these  agents  are  developed 
and  their  integration  into  the  virtual  world  is  important. 
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•  Scripting  -  this  involves  the  ability  of  being  able  to  record  and  playback  the 
interactions. 

•  Hypermedia  -  this  considers  the  possibility  of  allowing  the  addition  of  audio  and 
still  images  into  the  3D  geometrically  described  virtual  world. 

As  can  be  seen,  there  are  many  diverse  software  aspects  d\at  need  to  be  considered 
when  developing  a  complete  virtual  world. 


5.6  Toolkits 

5.6.1  Types 

In  the  above  sections,  basic  computer  graphics  terms,  object  representatioirs  and 
behaviours,  and  building  virtual  worlds  have  been  discussed.  As  can  be  seen,  if 
wishing  to  build  from  scratch  a  virtual  environment  to  be  used  within  a  VR  system, 
there  are  many  issues  which  need  to  be  resolved.  An  alternative  method  of  building 
virtual  environments  is  to  use  software  products  or  toolkits  which  have  already  been 
developed.  This  section  discusses  some  of  these  toolkits. 

There  are  two  major  categories  for  the  available  VR  software;  Toolkits  and  authoring 
systems.  Toolkits  are  programming  libraries,  generally  in  C  or  C++  that  provide  a  set 
of  functions  with  which  a  skilled  programmer  can  create  VR  applications.  Authoring 
systems  are  complete  programs  with  graphical  interfaces  for  creating  worlds  without 
resorting  to  detailed  programming.  These  usually  include  some  sort  of  scripting 
language  in  which  to  describe  complex  actions  [Isdale  1993].  The  programming 
libraries  are  generally  more  flexible  than  the  authoring  systems,  but  you  must  be  a 
skilled  programmer  to  use  them  [Isdale  1993].  The  VR  Sourcebook  [1996]  provides  a 
good  source  of  information  of  commercial  companies. 

Modelling  toolkits  have  both  shape  libraries  (with  hold  parametric  definitions  of 
circles,  triangles,  squares,  arcs,  ellipses,  and  polygons)  and  object  library  (with  higher- 
level  shapes  such  as  cubes,  spheres,  cylinders)  [Vince  1995].  Also,  most  toolkits  have 
some  sort  of  control  language  which  is  used  to  support  all  aspects  of  interaction, 
animation  and  physical  (i.e.  real)  simulation  [Vince  1995].  Toolkits  are  extendable 
libraries  of  object  orientated  fxmctions.  Libraries  are  extendable  in  that  is  it  possible  for 
developers  to  write  application-specific  modules  and  still  use  the  simulation  kernel 
[Burdea  and  Coiffet  1994].  Some  VR  Toolkits  are  hardware  independent. 

The  aim  of  VR  toolkits  is  to  help  programmers  in  system  integration  and  application 
development.  These  are  extremely  useful  tools  and  can  cut  programming  time 
substantially.  As  these  toolkits  also  accept  CAD  3D  files  (dxf  and  others)  this 
represents  a  significant  reduction  of  development  time  by  importing  existing  3D  object 
databases.  Commercial  packages  support  a  large  variety  of  high-end  input/  output 
tools,  such  as  sensing  gloves  (DataGlove  and  CyberGlove),  advanced  graphics  and 
networking  for  multi-user  simulations  [Burdea  and  Coiffet  1994].  Non-commercial 
toolkits  support  less  expensive  input/ output  devices,  and  have  less  performance 
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graphics.  However,  they  are  useful  for  proof-of-coricept  in  initial  research,  as  weU  as 
teaching  the  fundamentals  of  VR  [Burdea  and  Coiffet  1994]. 

In  summary,  according  to  MosheU  and  Dimn-Roberts  [1994],  toolkits  are  useful  as 
starting  points  for  the  construction  of  customized  VR  systems.  They  provide  the 
following  capabilities: 

•  tracking  information  from  HMDs  and  other  devices  (such  as  gloves); 

•  stereoscopic  displays; 

•  accepting  CAD  models;  and 

•  the  ability  to  have  various  degrees  of  physical  modeling  and  detail  control. 

A  survey  of  VR  services  and  products  (both  hardware  and  software)  was  done  by  the 
AI  Expert  Joximal  in  both  1994  and  1995.  In  1994,  this  list  contained  a  total  of  14  pages, 
while  in  1995,  this  list  had  increased  28%  to  a  total  of  18  pages  [VR  Resources  1995]. 

5.6.2  Available  VR  Software 

The  following  contains  some  of  the  available  VR  software  listed  in  alphabetical  order: 

AVIARY:  This  system  was  developed  at  Manchester  University,  United  Kingdom  and 
is  available  on  Sxm  and  SGI  platforms.  It  is  a  multi-user  virtual  environment  composed 
of  loosely  connected  autonomous  objects  which  execute  concurrently  [Kelly  1994]. 

DIVE:  This  was  developed  at  the  Swedish  Institute  of  Computer  Science  and  is 
available  for  the  Sun  and  SGI  platforms.  It  is  an  experimental  interactive  platform 
which  provides  a  navigable  shared  3D  S5mthetic  environment  in  which  users  can  meet 
and  collaborate.  It  supports  the  development  of  televirtuality,  user  interfaces  and 
applications  [Hand  1995,  Kelly  1994]. 

dVS:  Developed  by  Division  Ltd  (UK)  for  the  SGI  and  IBM  workstation  platforms 
[Isdale  1993,  Kalawsky  1993,  KeUy  1994,  Vince  1995,  VR  Resources  1995].  It  is  an 
environment  which  has  actors  that  are  responsible  for  various  system  activities  -  e.g.  to 
support  collision  detection  and  audio.  Each  actor  is  responsible  for  a  different  function 
of  the  system.  An  actor  may  be  a  single  process  or  a  collection  of  processes  that  interact 
with  the  environment  database.  An  actor  is  created  to  simulate  a  component  of  the 
virtual  environment  or  to  act  as  an  interface  to  the  real  environment.  Each  actor  can  be 
freely  created  or  deleted  to  perform  a  specified  task  and  may  have  a  limited  life  span. 
The  cost  of  dVS  ranges  from  $55,000  to  $200,000  depending  upon  options  [Buying 
Guide  1996]. 

GVS:  Developed  by  Gemini  Technology  Corporation,  GVS  is  a  Generic  Visual  System. 
It  produces  an  integrated  virtual  environment  display  which  is  needed  to  create 
discrete  3D  objects  and  integrate  them  into  an  interactive  visual  display.  There  are 
several  ways  of  doing  this.  An  effective  approach  is  to  employ  a  3D  modeling  package 
to  create  the  virtual  objects  and  devise  a  means  of  integrating  these  into  a  virtual 
world.  GVS  aims  to  produce  a  real-time  visual  simulated  environment  with  minimal 
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levels  of  coding.  The  emphasis  here,  is  on  employing  high  level  tools  to  provide  a 
visual  system  that  is  connected  to  a  distributed  computing  environment.  When 
coupled  with  a  3D  modelling  tool,  it  is  powerful  software  environment  for  creating 
and  interacting  with  a  virtual  environment  [Kalawsky  1993]. 

Humanoid:  Developed  by  the  Swiss  Federal  Institute  of  Technology,  this  is  a  real  time 
and  parallel  system  for  the  simulation  of  virtual  humans  including  motion  control, 
deformations  and  object  grasping  [Kelly  1994]. 

Jack:  This  human  body  simulation  is  a  general  purpose  interactive  environment  for 
maniprdating  articulated  geometric  figiues.  Jack  has  the  following  aspects:  hands, 
reaching,  grasping,  collision  detection,  collision  avoidance,  walking  behaviour,  balance 
control,  field-of-view,  joint  torque  and  sensor  channels.  Jack  can  be  used  in  conjunction 
with  other  simulation  programs  and  can  be  measmed  for  fit,  reach,  movement  and  etc. 
[Vince  1995,  VR  Resources  1995]. 

MRToolkit:  Developed  by  the  University  of  Alberta,  this  is  a  free  programming  library 
(to  Universities  and  other  research  estabhshments)  for  UNIX  systems  [Hand  1995, 
Isdale  1993,  Kalawsky  1993].  This  Minimal  Reality  Toolkit  (i.e.  MRToolkit)  is  a 
subroutine  library  that  supports  the  development  of  virtual  environment  and  other 
forms  of  3D  interfaces.  It  provides  basic  services  that  are  required  to  produce  virtual 
envirorunent  user  interfaces,  as  weU  as  providing  support  for  various  devices  such  as 
the  Polhemus  Tracker  and  the  VPL  DataGlove.  MRToolkit  also  supports  the 
distribution  of  the  user  interfaces  over  multiple  workstations,  data  distributed  oyer 
several  workstations,  numerous  interactive  techniques  and  real-time  performance 
analysis  tools.  It  can  be  called  from  C  programs  on  both  SGI  and  DEC  equipment 
[Kalawsky  1993]. 

MultiGen:  This  is  an  interactive  tool  for  the  creation  and  editing  of  visual  system 
databases.  MultiGen  is  a  specialised  Computer  Aided  Design  (CAD)  system,  which 
emphasises  things  that  are  needed  in  real-time  simulation;  other  CAD  emphasis  is  on 
engineering  aspects  and  artistic  aspects  (texture  and  shading)  [Moshell  and  Durm- 
Roberts  1994].  It  is  available  on  several  platforms.  This  tool  has  become  a  standard  for 
the  flight  simulation  community.  The  ease  in  which  sophisticated  models  can  be 
developed  for  use  in  virtual  environments  offsets  its  high  cost.  MultiGen  also  allows  a 
user  to  incorporate  the  Defense  Mapping  Agency  (DMA)  Digital  Terrain  Elevation 
Data  (DTED). 

SmartScene:  Developed  by  MultiGen  Inc.,  this  is  an  innovative  rapid  protot3rping  tool 
which  can  use  a  helmet  and  gloves  to  interactively  constructed  a  virtual  environment 
using  intelligent  models  from  a  existing  palette  [Ml)miec  and  Mapes  1995,  SmartScene 
1995,  VR  News  1995].  This  product  is  described  below,  in  the  section  on  iimovative 
systems. 

VEOS:  Developed  by  the  Human  Interface  Technology  Laboratory  at  the  University  of 
Washington,  VEOS  is  a  UNIX  based  operating  system  shell  which  provides  a 
comprehensive  and  imified  management  faciHty  for  generation  of,  interaction  with 


33 


DSTO-TR-0501 


and  maintenance  of  virtual  environment  [Isdale  1993,  Kalawsky  1993,  Kelly  1994].  It 
provides  a  user  level  framework  for  prototyping  distributed  applications.  The 
underlying  design  of  VEOS  was  to  create  a  mechanism  for  specifying  the  tasks  and 
assigning  computational  resomces  in  a  distributed  design;  it  is  written  in  LISP 
[Kalawsky  1993].  A  VEOS  application  can  be  broken  down  into  distinct  processes 
which  are  as  self-rehant  as  possible.  VEOS  is  platform  independent  and  has  run  on 
SGIs,  Suns  and  DECs. 

VRT:  Developed  by  Superscape,  VRT  has  dynamic  features  that  mimic  real  objects 
(e.g.  gravity,  fuel,  climbing,  falling,  etc)  [Vince  1995].  VRT  consists  of  several  editors  - 
the  two  important  ones  are  Shape  editor  which  provide  interactive  tools  for  building 
polygonal  objects  and  World  editor  which  organises  the  construction  of  virtual  worlds 
from  these  objects  [Vince  1995,  VR  Resources  1995].  This  product  costs  $5000  [Buying 
Guide  1996]. 

World  Tool  Kit  (WTK):  Developed  by  SenseS,  WTK  is  probably  the  most  widely  used 
product  of  this  type.  It  runs  on  a  wide  variety  of  platforms  from  high  end  PCs  to  high 
end  SGI  boxes.  It  has  won  several  awards  for  excellence  [Hand  1995,  Isdale  1993, 
Kalawsky  1993,  Pimentel  and  Teixeira  1993,  VR  Resources  1995].  WTK  costs  $795  to 
$12,500  depending  upon  options. 

WorldToolKit  (WTK)  is  a  library  of  C  ftmctions  that  can  be  used  to  write  a  C  program 
for  developing  and  interacting  with  virtual  worlds.  The  user  writes  a  C  program 
referencing  the  functions,  which  then  becomes  a  WTK  application  [Vince  1995].  WTK 
contains  over  400  functions  and  a  powerful  simulation  manager  that  allows  rapid 
prototyping  of  custom  applications.  3D  models  created  with  AutoCAD  or  any  3D 
modeller  that  generates  dxf  files  can  be  read  as  objects  [Pimentel  and  Teixeira  1993]. 
From  a  user's  point  of  view,  WTK  provides  a  complete  application  development 
environment  for  s5mthesising  virtual  environment.  One  of  its  most  attractive  feature  is 
its  hardware  independence. 

As  WTK  is  structured  in  an  object  oriented  naming  manner,  it  is  easier  to  maintain 
visibility  at  a  programming  level  of  what  is  taking  place.  Its  main  classes  are  universe, 
object,  polygon,  vertex,  path,  sensor,  viewpoint,  tight  source,  portal  and  animation. 
The  universe  is  the  uppermost  class  and  only  one  can  be  active  within  the  WTK  virtual 
environment.  A  umverse  is  built  of  objects,  including  sensors  (external  devices  that 
provide  input  into  tiie  computer  to  control  the  position  and  orientation  of  objects  such 
as  mouse,  tracker,  etc),  tights,  animation  sequences,  portals,  viewpoints,  graphical 
objects,  serial  ports  and  others  [Kalawsky  1993]. 

The  user  of  WTK  has  to  be  a  competent  C  programmer.  However,  this  package  is  an 
excellent  systems  for  those  who  want  control  over  the  creation  of  virtual  environment 
at  a  detailed  level  and  at  a  modest  price.  An  effective  method  is  to  develop  the 
application  on  a  low  cost  PC  and  transfer  the  application  to  a  high  end  workstation 
[Kalawsky  1993]. 


34 


DSTO-TR-0501 


WorldUp;  Developed  by  SenseS,  this  toolkit  is  based  on  SenseS's  WorldToolKit  (WTK) 
discussed  above.  WorldUp  is  a  VR  and  visual  simulation  development  tool  that  aUows 
interactive  world  creation  without  the  need  for  C  programming.  WorldUp's  interactive 
development  environment  allows  users  to  modify  real  time  object  behaviour  and  scene 
characteristics  while  the  simulation  is  still  running.  Effects  are  seen  straight  away  - 
there  is  no  need  to  stop  and  re-compile.  Also  this  toolkit  includes  tools  for  modelling, 
building  and  animating  objects  and  environments.  Also,  3D  models  created  using 
many  CAD  packages  can  be  imported  [VR  News  1995]. 

3D  Builder:  Even  though  this  is  not  specifically  a  VR  product,  such  products  will 
influence  the  creation  of  the  virtual  environment.  3D  Builder  builds  3D  models  of  real 
world  data  from  photographs.  It  covers  a  wide  range  of  custom  modelling  situations, 
from  a  single  picture  to  projects  (such  as  a  large  building  or  a  city  neighbourhood)  that 
require  a  coordinated  effort  using  dozens  of  pictures.  A  snap-to-grid  axis  is  included, 
as  is  the  ability  to  view  the  model  both  as  entered  and  as  calculated.  Minimal,  flexible 
measurements  are  needed,  and  in  many  cases,  where  there  are  natural  x,  y  and  z  cues, 
no  measurements  may  be  necessary  to  create  a  3D  model.  Using  a  comprehensive 
math  solver,  3D  builder  is  able  to  combine  information  from  a  large  number  of  photos 
of  large,  complicated  objects,  extract  information  and  merge  it  all  together  into  a  single 
3D  model  ready  to  export  to  the  target  rendering  program.  This  tool  offers  a  variety  of 
input  formats  including  Photo  CD,  TIFF,  JPEG,  BMP,  and  Sun  Raster.  The  tool  can 
produce  3D  dxf,  IGES,  3D  studio  and  Inventor  VRML  files  [VR  News  1995]. 

The  VR  resoxirces  guide  [1995]  listed  several  other  interesting  items,  as  listed  below: 

•  CM  research  -  add  sensation  of  temperature. 

•  Coryphaeus  Software  -  developed  the  Designer's  workbench  which  can  be  used  as 
an  environment  for  creating  and  editing  dynamic  instrument  displays  and  3D  VR 
scenes. 

•  Digital  Media  Group  -  uses  StereoSynthesis  which  is  a  process  which  converts 
standard  2D  film,  video  or  still  images  to  true  stereoscopic  3D  format. 

•  SimGraphics  -  computer  generated  characters. 

•  Thtmderseat  VR  chair  -  originally  developed  for  air  combat  trainers,  this  chair  uses 
sound  output  to  generate  realistic  vibratory  sensations. 

•  Virtual  Presence  -  uses  the  Genesis  World  Builder  which  is  an  interactive 
prototyping  tool  for  use  with  WorldToolKit  VR  software.  This  tool  gives  the  user 
the  ability  to  manipulate  objects,  polygons,  lights,  sensors,  path  and  so  on  in  real¬ 
time. 

•  Virtus  corp  -  uses  Virtus  VR  which  uses  a  drag-and-drop  approach  to  world 
creation.  Basic  3D  shapes  help  build  complex  objects  and  structures.  The  tool 
provides  the  construction  and  navigation  tools,  galleries  of  3D  objects  and  pre-built 
scenes  (PC  and  Mac  only). 

•  Xtensory  -  produces  tactile  systems. 
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5.7  Final  Comments 

Kalawsky  [1993]  states  that  most  aspects  of  VR  rely  on  an  underlying  database 
standard  on  which  to  represent  the  objects  in  the  virtud  environment.  In  this  manner, 
virtual  environments  are  similar  to  Computer  Aided  Design  (CAD).  However, 
complete  VR  systems  require  a  lot  more  to  describe  the  environment.  Of  course  the 
obvious  geometrical  and  spatial  relationship  of  objects  are  required.  However,  in  VR, 
additional  object  information  is  required;  information  such  as  the  texture,  feel  and 
compliance  of  an  object  needs  to  be  specified.  Also  parameters,  such  as  behaviour  and 
interactions  between  objects,  must  be  included.  This  must  include  the  responses  to 
external  events  or  stimuli  (such  as  collisions)  and  also  should  include  the  weight  and 
feel  of  an  object.  Pimentel  and  Tebceira  [1993]  states  that  this  is  the  fundamental 
problem  with  all  3D  modelling;  i.e.  the  developers  of  3D  models  are  only  concerned 
with  the  visual  representation  of  the  objects.  However,  the  object's  physical  properties, 
weight,  center  of  mass,  composition  of,  sound  with  contact  witii  another  surface, 
surface  spongy  or  hard,  rough  or  smooth  are  just  some  of  the  attributes  which  need  to 
be  included  as  part  of  the  database  for  a  VR  system. 


6.  Innovative  Systems 


In  this  section  several  innovative  systems  are  described.  The  first  two,  SmartScene  and 
Polyshop,  are  interesting  in  that  they  used  VR  techniques  to  build  terrain  and/or 
visual  databases.  The  Responsive  Workbench  and  the  Virtual  Workbench,  although 
quite  similar,  each  have  a  different  focus.  These  are  systems  which  provide  an 
workbench  from  which  to  work  on.  Finally,  the  CAVE  system  is  described.  The  CAVE 
uses  a  room  as  its  work  space  area.  Imagery  is  displayed  on  three  walls  and  tire  floor  of 
this  room. 


6.1  SmartScene 

The  basis  of  SmartScene  (developed  by  MxtitiGen)  is  that  by  placing  identifiable  pre¬ 
built  elements,  such  as  a  building  or  a  chair,  in  a  scene  is  quicker  and  more  efficient 
than  having  to  build  the  elements  from  scratch.  This  obviously  requires  the  availability 
of  a  comprehensive  model  library  so  that  an  appropriate  model  is  available  most  of  the 
time  [Mlyniec  and  Mapes  1995].  SmartScene  also  uses  ModelTime  behaviours  which 
are  part  of  the  models  themselves  and  can  be  can  be  thought  of  as  intelligent, 
three-dimensional  "snap-to-grid"  constraints  [Mlyniec  and  Mapes  1995].  These 
behaviours  are  used  to  tell  objects  how  to  behave  when  they  are  being  assembled  with 
other  such  objects  [SmartScene  1995,  VR  News  1995]. 

ModelTime  behaviours  bridge  the  gap  between  rough  positioning  and  precision 
alignment  [Mlyniec  and  Mapes  1995].  With  no  knowledge  of  the  high-level 
relationships  between  objects  in  the  scene,  the  snapping  of  vertices  to  other  vertices  in 
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the  scene  can  be  accomplished.  Edge-to-edge  and  face-to-face  alignment  are  also 
useful.  These  higher-level  smart  snapping  relationships  are  stored  with  an  object.  So, 
for  example,  the  second  story  of  a  budding  can  seek  out  its  first  floor,  ignoring  the 
terrain,  houses,  and  even  other  multi-story  buildings  of  incompatible  type  [Mlyniec 
and  Mapes  1995,  SmartScene  1995,  VR  News  1995].  The  result  is  a  collection  of 
seemingly  magnetic  objects  which  attract  and  repel  in  an  intuitive  fashion  [Mlyruec 
and  Mapes  1995].  Thus,  it  becomes  a  lot  simpler  and  more  intuitive  to  snap  standard 
scene  components  together  (particularly  where  the  builder  is  disadvantaged  by  an 
HMD)  because  the  precision  alignments  can  be  made  to  happen  automatically,  via  the 
simulated  magnetic  attraction  or  repulsion  between  objects,  vertices  and  faces  [VR 
News  1995]. 

Scaling  and  stretching  behaviours  can  also  be  triggered  by  the  proximity  of  compatible 
objects  during  the  construction  of  virtual  environment.  When  an  object  is  stretched,  it 
may  not  be  enough  to  uniformly  grow  it  in  tihe  direction  in  which  it  is  stretching.  A 
stretch  threshold  is  used  to  invoke  a  certain  behaviour  each  time  a  certain  stretch 
condition  is  met.  For  instance,  stretching  a  building  could  have  the  effect  of  adding 
stories  to  it  by  repeating  the  geometry  or  texture  appropriately,  instead  of  simply 
stretching  the  one  floor  [Mlyniec  and  Mapes  1995].  This  would  seem  to  require  some 
form  of  'real'  measurement  to  be  incorporated. 

During  construction,  the  physical  extent  of  an  object  can  be  used  to  support  collision 
modelling.  Gravity  can  also  be  used  to  manipulate  objects  in  a  scene  if  the  objects  are 
told  to  act  under  that  force  during  a  modelling  session.  Objects  with  mass,  moment  of 
inertia,  flexibility,  and  so  on  can  further  enable  the  builder  to  treat  the  assembly  of  a 
synthetic  scene  as  the  everyday  manipulation  of  famihar  elements  [hllyniec  and 
Mapes  1995]. 

The  rapid  prototyping  technique  discussed  so  far  has  an  inherent  drawback;  it  is 
difficult  to  create  scenes  that  do  not  look  repetitive  even  with  an  extensive  model 
library  [Mlyniec  and  Mapes  1995].  By  building  smart  palettes  into  the  models,  that  is 
attribute  palettes  designed  specifically  for  the  object,  variety  can  be  attained  through 
multiple  choice  [Mlyniec  and  Mapes  1995]. 

ModelTime  behaviours  is  well  suited  to  immersion  [Mlyniec  and  Mapes  1995].  A 
stereoscopic  3D  HMD  immerses  the  builder  in  a  3D  visual  work  space  and  a  two- 
handed  Fakespace  Pinch  Glove  interface  puts  3D  widgets,  model  components,  texture 
and  colour  in  the  pahn  of  the  hand  [SmartScene  1995,  VR  News  1995].  Natural  hand 
gestures  can  then  be  used  to  grab  and  manipulate  the  scene  components  with  one  or 
both  hands.  The  task  of  positioning,  scaling  and  rotating  are  done  in  a  single,  natural 
hand  gesture.  The  advantage  of  this  system  is  that  it  exploits  the  natural  human  usage 
of  their  hands.  Figure  10  has  an  interesting  view  of  SmartScene. 
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Figure  10:  The  SmartScene  tool  from  MultiGen  (adapted  from  [SmartScene  1995]). 


6.2  PolyShop 

PolyShop  is  a  research  project  at  the  Institute  of  Simulation  and  Training  (1ST)  based  in 
Florida,  USA  [Abel  et  al  1995].  It  is  focussed  on  the  construction  of  3D  visual  database 
using  VR.  Its  objective  is  to  examine  and  develop  a  networked  multi-user,  3D 
immersive  user  interface  for  the  rapid  construction  of  urban  databases  [Abel  et  al 
1996].  The  network's  purpose  is  to  allow  multiple  participants  to  work  on  the  same 
database  simultaneously  at  remote  locations.  VR  technology  provides  the  3D 
immersion  and  the  interactive  stereoscopic  vision.  This  enables  the  modellers  to  have  a 
more  intuitive  spatial  orientation  and  perspective  of  the  objects  they  are  building. 

The  modeller  sits  at  a  desk.  The  top  of  the  desk  is  reproduced  in  the  virtual  world  with 
virtual  buttons,  sliders  and  controls  added  to  it.  For  geometric  modeling,  the  user  sits 
at  a  drafting  table  style  desk  constructed  to  minimise  interference  with  the  trackers. 
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The  physical  and  virtual  desktop  are  calibrated  so  that  the  virtual  one  mirrors  the  real 
one.  Different  tools  are  also  modelled.  The  modeller  is  immersed  with  a  HMD  and 
gloves  for  object  and  tool  manipulation.  Sound  feedback  is  used  to  make  interactions 
better;  soxmd  is  used  for  indicating  the  proximity  of  an  object. 

The  main  aim  of  PolyShop  is  to  have  a  shared  virtual  environment  where  several 
modellers  at  different  locations  can  collaborate  simultaneously  on  same  graphical 
database.  PolyShop  uses  enhanced  Distributed  Interactive  Simulation  (DIS)  protocols 
for  communication  and  modelling  a  particular  world.  Standard  DIS  is  not  able  to  send 
dynamic  data  across  network.  The  developers  of  PolyShop  have  proposed  a  geometric 
state  PDU  (basic  database  changes  over  DIS  network  in  real-time)  [Abel  et  al  1995]. 

u^siupiilation  of  the  virtual  objects  and  control  is  done  with  a  two  handed 
interface,  based  on  ChordGloves.  These  gloves  were  developed  at  the  1ST.  These 
ChordGloves  are  used  to  manipulate  the  objects  in  PolyShop's  3D  virtual  space.  Glove 
manipulations  are  an  intuitive  interface  for  humans  [Abel  et  al  1995].  Also  different 
hand  gestures  can  be  used  to  perform  different  actions. 

The  modeller  grabs  an  object  in  3D  using  both  gloves  and  then  selects  the  desired 
action  (e.g.  scale,  stretch)  and  then  shapes  it  in  the  required  way.  Texture  and  colour 
can  be  added  to  the  objects,  as  well  as  glueing  objects  together.  From  this,  a  complex 
database  can  be  constructed.  This  is  a  powerful  3D  modelling  interface  [Abel  et  al 
1995]. 

A  Polhemus  FasTrak  trackers  with  three  sensors  is  used  to  track  the  head  and  both 
hands.  A  low  cost  HMD  is  used  for  the  immersive  stereo  display.  Sound  feedback  and 
voice  input  enhances  interactions  and  adds  versatihty  to  the  tool.  However,  both 
require  dedicated  PC  boards.  For  the  sound  feedback,  specific  events  give  specific 
soxmds  in  order  to  provide  additional  feedback.  Voice  recognition  module  receives 
input  from  a  microphone  and  produces  an  encoded  command  (e.g.  stretch,  colour,  etc). 

The  software  of  PolyShop  is  written  on  top  of  WorldToolKit  (WTK  -  see  above).  WTK 
handles  the  simulation  loop,  sensors  and  basic  object  organisation.  The  software  has 
two  distinct  layers  [Abel  et  al  1995]: 

•  First  layer  has  the  following  characteristics  (a)  unique  VR  software  which  includes  a 
manipulation  metaphor,  (b)  direct  mapping  between  virtual  and  physical  worlds, 
(c)  fatigue  is  decreased  for  reaching  objects  by  aUowing  an  appHcation  to  be  built 
around  a  physical  desktop  and  (d)  has  precision  object  ahgnment  operations. 

•  Second  layer  is  the  widget  toolkit  layer  which  is  totally  dependant  upon  the  first  or 
VR  baseline  layer.  This  toolkit  has  an  array  of  useful  tools,  including  a  chooser  for 
bringing  models  into  the  world,  light,  colour  changes,  menus  and  trashcans. 

PolyShop  can  be  run  on  a  stand  alone  workstation  or  networked  allowing  multiple 
participants  to  interact  with  same  database  through  Ethernet  serial  communications. 
The  initial  product  was  prototyped  on  a  PC;  it  is  currently  being  ported  to  SGls 
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workstations.  It  can  also  be  run  with  different  hardware  configurations  and  a  range  of 
operation  modes.  The  base  level  is  a  keyboard  mode  with  a  single  window,  while  an 
advanced  level  uses  full  immersion  with  a  HMD  and  gloves  [Abel  et  al  1995]. 


6.3  Responsive  Workbench 

The  "Responsive  Workbench"  concept  is  a  system  where  the  user  no  longer  experiences 
simulations  of  the  world  on  the  computer.  Instead,  the  computer  is  integrated  into  the 
user's  world  in  a  way  which  is  not  seen  by  the  user.  To  enhance  the  illusion,  common 
everyday  objects  and  activities  become  the  inputs  and  outputs  for  this  environment 
[Bleicher  and  Heiden  1995].  This  concept  was  pioneered  by  W.  Kruger  from  Germany 
[Bleicher  and  Heiden  1995].  The  responsive  workbench  has  been  duplicated  at  tire 
Naval  Research  Laboratory,  USA  [Durbin  1995]. 

The  Responsive  Workbench  is  a  3D  interactive  graphics  system  where  computer¬ 
generated  stereoscopic  images  are  projected  onto  a  tabletop  via  a  projector-and-mirrors 
system.  The  user  of  the  workbench  wears  shutter  glasses  to  observe  the  3D  effect.  A 
tracking  system  tracks  the  user's  head  and  hand  positions  to  allow  interaction  with 
objects  in  the  tabletop  environment  [Bleicher  and  Heiden  1995,  Durbin  1995,  Kruger  et 
al  1995].  Other  participants  can  also  interact  in  the  shared  virtual  environment  and 
shared  physical  place;  however,  only  one  participant  is  tracked  [Durbin  1995]. 

Virtual  objects  and  control  tools  are  located  on  a  real  workbench.  The  objects  displayed 
as  computer-generated  stereoscopic  images  are  projected  onto  the  surface  of  a  table. 
This  can  correspond  to  the  actual  work  situation  of,  for  example,  an  architect's  office  or 
a  surgeon's  operating  table.  A  guide  uses  the  virtual  working  environment  while 
several  observers  can  watch  events  through  shutter  glasses.  The  several  observers  can 
work  together  locally  or  use  global  communication  networks  [Bleicher  and  Heiden 
1995]. 

The  Responsive  Workbench's  physical  hardware  consists  of  a  projector,  a  mirror,  a 
translucent  table  top  and  the  LCD  shutter  glasses  are  StereoGraphics'  CrystalEyes 
[Durbin  1995];  see  Figure  11.  The  German  version  of  the  workbench  uses  a  CyberGlove 
and  the  Polhemus'  Fastrak  sensors  for  head  and  hands  tracking  [Kruger  et  al  1995], 
while  at  the  Naval  Research  Laboratory,  locaUy  developed  gloves  and  the  PoUiemus' 
3SPACE  tracker  system  are  used  [Durbin  1995].  The  software  used  is  the  SGI  Graphics 
Library  (GL),  the  Performer  software.  Hark  voice  recognition  system  and  the  SVE 
software  package  [Kruger  et  al  1995]. 
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Examination  of  the  blood  flow 
in  a  human  heart 


Stereoscopic  display  of  virtual 
objects  on  a  table 


Cooperative  work  scenario  of  a 
physician  and  a  student 


In  the  scenario,  a  physician  picks 
up  a  bone  with  the  glove  device 


Figure  11;  The  Responsive  Workbench  and  some  of  its  usages 
[adapted  from  Krueger  et  al  1995]. 
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The  following  applications  have  been  developed  for  use  with  the  Responsive 

Workbench: 

•  non-sequential  medical  training  (the  scenario  is  based  on  a  real  sized  model  of  a 
patient  who  could  be  examined  in  any  detail  through  the  zooming  operation;  also 
important  in  this  application  is  the  dynamic  aspects,  like  the  beating  heart  and  the 
blood  flow  inside  of  the  body); 

•  surgery  planning; 

•  the  fluid  dynamics  visualisation  in  a  virtual  windtunnel;  and 

•  the  three-dimensional  simulation  of  design  in  architecture  and  landscape  planning. 

Figures  11  shows  some  of  these  applications  [Kruger  et  al  1995]. 


6.4  Virtual  Workbench 

Although  the  system  known  as  the  Virtual  Workbench  is  similar  to  the  Responsive 
Workbench  described  above,  the  focus  of  this  Workbench  is  quite  different.  It  main 
goal  is  to  produce  a  work  environment  in  which  delicate  work  can  be  performed  for 
long  hours  without  any  strain  [Poston  and  Serra  1994,  1996].  This  is  achieved  by 
embedding  the  virtual  world  in  the  user's  natural  work  volume. 

The  natural  work  volume  is  represented  by  a  region  of  a  foot  or  so  in  front  of  the  eyes; 
this  is  for  precise,  comfortable  depth  perception  and  within  easy  reach  of  both  hands 
[Poston  and  Serra  1996].  The  human's  hand-eye  coordination  ability  is  used  to  achieve 
dexterity  in  this  natural  work  volume.  Here,  the  user  both  feels  where  something  is 
(through  their  own  hand  holding  something)  and  seeing  it  in  the  virtual  space.  In  this 
case,  the  something  is  a  generalized  tool  handle.  However,  what  the  user  sees  in  tiie 
virtual  space  is  a  virtual  tool  (such  as  a  cutter  or  pencil),  which  the  software  creates. 
Thus,  the  user  feels  where  the  hand  is  and  where  the  physical  tool  is,  and  this  matches 
the  virtual  tool  in  the  virtual  space  [Poston  and  Serra  1994].  The  caHbration  between 
the  tool  handle  and  the  virtual  tool  is  obviously  important  [Poston  and  Serra  1994, 
1996].  It  is  also  interesting  to  note  that  only  the  handle  is  modelled;  the  actual  hand 
holding  the  handle  is  not  [Poston  and  Serra  1996]. 


The  Virtual  Workbench,  shown  in  Figure  12,  consists  of  a  computer  screen,  a  pair  of 
stereo  glasses  (Crystal  Shutter  Glasses),  a  mirror  and  one  or  more  3D  position  and 
orientation  sensors  which  are  on  some  sort  of  handle  (i.e.  are  grippable)  [Poston  and 
Serra  1994,  1996];  an  example  of  such  a  sensor  is  the  Immersion  Probe  .  Haptic 
devices  to  provide  force  feedback  are  also  being  considered.  The  virtual  image  within 
the  work  volume  is  perceived  by  the  user  by  looking  through  both  the  mirror  and  the 
glasses.  Currently,  the  Virtual  Workbench  is  based  on  a  SGI  ReaUty  Engine;  however,  a 
version  based  on  a  high-end  PC  is  currently  being  developed. 


42 


DSTO-TR-0501 


Figure  12:  The  Virtual  Workbench  [adapted  from  Poston  and  Serra  1994]. 
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The  Virtual  Workbench  supports  both  fixed  and  head-tracked  viewpoints  [Poston  and 
Serra  1996].  Fixed  viewpoints  are  useful  for  objects  within  the  natural  work  voltune. 
However,  this  is  not  sufficient  for  judging  the  proximity  of  an  object  which  is 
perpendicular  to  the  user.  A  head-tracked  viewpoint  is  useful  here  and  in  high 
precision  work  where  a  sHght  movement  of  the  head  can  help  estimate  the  proximity 
of  an  object.  In  order  to  evaluate  the  performance  and  ergonomics  of  this  system,  a 
dexterity  test  was  developed.  This  showed  that  stereo  vision  and  head-tracking  are 
important  [Poston  and  Serra  1996, 1994]. 

To  date,  this  Virtual  Workbench  has  been  primarily  used  in  medical  appHcations. 
Some  of  these  are  hsted  below: 

•  visuaHsation  tools  for  diagnostics  [Poston  and  Serra  1996];, 

•  contour  editor  showing  the  heart  walls  [Poston  and  Serra  199];, 

•  tube  finder  for  identifying  blood  arteries  [Poston  and  Serra  1996,  Poston  et  al  1995]; 
and 

•  heart  disease  assessment  [Solaiyappan  et  al  1996]. 


6.5  CAVE 

The  main  objective  of  CAVE  (Cave  Automatic  Virtual  Environment)  is  to  develop  a 
visuahzsation  tool  which  can  be  used  by  scientists  [Cruz-Neira  et  al.  1993a].  Also,  the 
developers  of  the  CAVE  wanted  to  illustrate  that  the  correct  projection  of  the  imagery 
on  large  screens  can  also  create  a  VR  experience.  CAVE  stresses  that  projection 
technology  achieves  a  system  that  matches  the  quality  of  the  workstation  in  terms  of 
resolution,  colour  and  flicker-free  stereo  [Cruz-Neira  et  al.  1993b]. 

CAVE  uses  multiple  large  projection  displays  to  create  a  room  in  which  the  viewer(s) 
stand  [Isdale  1993].  CAVE  uses  stereoscopic  video  projectors  to  display  images  on 
three  surroxmding  walls  and  on  the  floor.  Projectors  display  fuU-colour  workstation 
fields  (i.e.  1280x512)  at  120Hz  onto  the  screens  [Cruz-Neira  et  al.  1993b],  The  use  of 
multiple  projection  systems  can  provide  immersive  displays  that  cover  the  entire 
visual  field  with  many  pixels,  but  increasing  the  number  of  screens  also  increases  the 
requirements  for  computing  graphics  [Roman  1995].  The  illusion  of  immersion  is  thus 
created  by  projecting  stereoscopic  computer  graphics  into  a  cube  composed  of  display 
screens  that  almost  completely  surroimd  the  viewer  [Cruz-Neira  et  al.  1993a].  The 
CAVE  and  its  arrangement  of  the  projectors  is  shown  in  Figure  13  [Lewis  1994]. 

The  participants  wear  glasses  with  LCD  stereo  shutters  to  view  the  3D  images  in  the 
CAVE.  Multiple  participants  can  be  in  the  CAVE  at  any  time.  However,  only  one 
participant  is  tracked  and  the  3D  images  are  referenced  to  that  person's  position 
[Roman  1995,  Kelly  1994,  Vince  1995].  That  person's  head  and  hands  are  tracked  with 
Polhemus  or  Ascension  devices.  Four  SGI  high-end  workstations  (i.e.  Onyx)  create  the 
imagery  (one  for  each  screen). 
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Figure  13:  The  CAVE  architecture  with  the  foxjr  projectors 
(adapted  from  [Lewis  1995]). 


The  CAVE  blends  both  real  and  virtual  objects  in  the  same  space.  People  in  the  CAVE 
have  an  un-occluded  view  of  their  own  bodies  as  it  interacts  with  the  virtual  world 
and  the  objects  within  it.  They  are  also  aware  of  the  real  surroundings.  This  removes 
the  possibility  (and  fear!)  of  walking  into  a  wall.  So,  the  CAVE  is  an  non-instrusive 
environment  where  the  real  and  virtual  worlds  are  naturally  combined  [Cruz-Neira  et 
al.  1993a]. 

Some  of  the  goals  of  CAVE's  engineering  effort  include  [Cruz-Neira  et  al.  1993b]: 

•  desire  for  high  resolution  colour  images  without  geometric  distortion; 

•  less  sensitivity  to  head-rotation  induced  errors;  and 

•  ability  to  mix  VR  imagery  with  real  devices  (such  as  a  human's  hand). 

CAVE  has  primarily  been  used  for  exploring  visualisation  of  precomputed  databases 
[Cruz-Neira  et  al.  1993b,  Vince  1995].  For  instance,  the  CAVE  has  been  used  to 
visuahse  a  human  brain  model  [Boman  1994].  The  use  of  the  CAVE  in  medical  research 
has  great  potential  as  it  can  translate  what  is  seen  in  a  microscope  into  a  larger-than- 
life  representation  [Lewis  1995].  Other  appHcations  include  [Cruz-Neira  et  al.  1993a]: 

•  architectural  walk-through; 

•  visualisation  of  the  molecular  dynamics  of  cancer; 

•  exploration  of  the  cosmos; 
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•  regional  scale  weather  display;  and 

•  modeling  superconductors  on  massively  parallel  computers. 

The  shortcomings  of  the  CAVE  are  as  follows  [Cruz-Neira  et  al.  1993b]: 

•  large  and  expensive; 

•  six  screens  would  have  been  better,  but  entry  and  exit  would  be  a  problem; 

•  Hght  reflection  from  various  screens;  and 

•  directional  sound  is  not  achieved  due  to  reflections  off  the  screens. 

The  developers  of  CAVE  believe  that  it  has  great  potential.  They  have  achieved  their 
goal  of  producing  a  large  field-of-view,  creating  high  resolution  full-colour  images  and 
allowing  multi-person  presentation  format.  A  smaller,  less  expensive  version  of  the 
CAVE,  called  ImersaDesk,  is  currently  being  developed  [Lewis  1995]. 


7.  Applications 

Jacobson  [1991]  states  that  VR  can  be  used  in  any  commercial  field  where  people  use 
computers  to  store,  manage,  manipulate,  analyse,  present  and  want  to  understand 
complex  data.  As  such,  VR  has  numerous  appHcations.  The  Japanese  have  already 
employed  immersive  systems  for  showing  products  to  home-buyers  building  new 
kitchens  [Kahaner  1994].  Some  coimtries  in  Europe  have  used  VR  for  manufacturing 
and  designing  new  user  interfaces  [Encarnacao  and  Gobel  1994].  One  organisation  in 
the  United  States  is  using  VR  techniques  for  the  automatic  3D  construction  from  image 
sequences  [Kelly  1994].  Also,  the  VR  developing  hardware  and  software  can  enhance 
existing  simulator  systems.  In  particular,  the  rapid  prototyping  toolkits,  such  as 
WorldToolKit  and  SmartScene,  have  the  potential  of  improving  productivity.  NASA's 
1993  White  Paper  listed  several  applications  of  VR  [NuU  and  Jenkins  1993];  this 
document  highlights  many  applications  ranging  from  scientific  visualisations  to 
training  systems,  as  well  as  speculating  about  future  VR  applications. 

In  this  section,  a  brief  list  of  some  of  the  current  applications  in  VR  will  be  presented. 
Firstly,  VR  applications  in  general  will  be  presented.  After  this,  VR  applications  in 
defence  will  be  given. 


7.1  General  Applications 

Boman  [1994]  has  a  useful  taxonomy  of  classifying  VR  applications.  This  taxonomy  is 
used  below  to  classify  some  of  the  current  appUcations  of  VR: 


Education  and  training: 

•  training  astronauts  and  flight  controllers  on  extravehicular  activities  for  space 
missions  (weightlessness  is  simulated)  [Boman  1995]; 
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•  training  flight  crews  on  the  operation  of  the  Space  Shuttle's  Remote  Mampulator 
System  Robot  Arm  [Boman  1995]; 

•  virtual  science  laboratory  where  the  student  can  control  gravity,  friction  and  time 
[Boman  1995];  and 

•  training  disabled  children  to  control  motorised  wheelchairs  [Boman  1995] . 

Design  and  testing: 

•  testing  the  visibility  of  different  tractor  designs  [Boman  1995]; 

•  automobile  manufacturers  testing  new  dashboards  [Boman  1995]; 

•  testing  new  aircraft  designs  [Boman  1995]; 

•  industrial  security  systems  [Nomura  1994]; 

•  architectural  walk-throughs  [Encamacao  and  Gobel  1994,  MosheU  and  Dunn- 
Roberts  1994]; 

•  software  tools  [Null  and  Jenkins  1993]; 

•  driving  simulations  [Bayarri  et  al  1994]; 

•  advanced  interfaces  [Encamacao  and  Gobel  1994]; 

•  rapid  prototying  of  terrain  databases  [Polis  et  al  1995];  and 

•  manufacturing  (i.e.  designing  new  products]  [Nomura  1994]. 

Information  visualisation: 

•  scientific  visualisation  [Bryson  1996,  Null  and  Jenkins  1993]; 

•  computational  fluid  dynamics  (i.e.  Virtual  Wind  Tunnel)  for  air  flow  over  aircraft 
wings  [MosheU  and  Dunn-Roberts  1994,  Jacobson  1991]; 

•  planetary  exploration  [NuU  and  Jenkins  1993]; 

•  visual  analysis  of  software  which  automaticaUy  control  complex  power  networks 
[Boman  1995];  and 

•  viewing  unages  from  a  scanning,  tunneUing  microscope  [Boman  1995] . 

Medicine: 

•  ultrasoxmd  image  superimposed  on  Hve  video  (X-ray  vision)  [Boman  1995]; 

•  providing  greater  interactivity  to  people  with  severe  disability  through  computer 
interfaces  that  use  eye  movements  and  muscle  potentials  [Boman  1995]; 

•  virtual  glass  elevator  to  treat  acrophobia  [Boman  1995]; 

•  robot  assisted  surgical  system; 

•  planning  of  radiation  therapy  [MosheU  and  Dunn-Roberts  1994];  and 

•  health  care  [Nomura  1994,  NuU  and  Jenkins  1993]. 

Other  cireas  include  the  appUcations  in  augmented  reality  (where  the  computer 
generated  imagery  is  combined  with  the  view  of  the  real  world)  and  telepresence 
(where  one  can  remotely  manipulate  equipment  at  a  remote  site)  [Boman  1995, 
MosheU  and  Dunn-Roberts  1994]. 
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7.2  Defence  Applications 

In  defence,  VR  has  great  potential.  According  to  Baumann  [1995],  there  are  three  types 
of  mihtary  traming  and  apphcatioris: 

•  simrdation  of  reality; 

•  extension  of  human  senses  through  telepresence;  and 

•  information  enhancers  through  augmented  reality. 

In  the  area  of  military  simulations,  multiple  monitors  are  used  to  simulate  the  entire 
field-of-view  from  an  aircraft  cockpit  [Baumann  1995].  This  is  a  simulation  of  reality 
type  of  appHcation.  This  type  of  application  is  commonly  used  for  transport  aircraft. 
Fighter  aircraft  simulations,  on  the  other  hand,  need  180  degrees  both  in  horizontal 
and  vertical.  A  common  configuration  has  a  cockpit  placed  in  tiie  center  of  a  domed 
room,  with  virtual  images  projected  on  the  inside  surface  of  the  dome.  An  example  of 
such  a  configuration  is  the  Air  Operations  Simulation  Centre  (AOSC)  partial  dome 
[Feik  and  Mason  1993a,  Feik  and  Mason  1993b].  In  addition.  Distributed  Interactive 
Simulation  (DIS)  communication  protocols  can  be  used  to  connect  several  simxilators. 
This  type  of  configuration  can  be  used  for  training  as  well  as  developing  and  testing 
new  combat  strategies  and  tactics. 

Telepresence  can  be  defined  as  a  medium  that  gives  a  person  the  sense  of  being  placed 
physically  within  remote,  computer-created  scenes  [Larijani  1994].  This  is  extremely 
useful  in  mihtary  appHcations  because  the  exposure  to  hazards  is  reduced.  In  addition, 
stealth  is  increased  dramatically.  Smart  vehicles  and  remotely-piloted  vehicles  were 
developed  for  this  [Baumann  1995]. 

Augmented  reahty  is  a  system  which  combines  both  real  and  virtual  views.  In  the 
dynamic  combat  environment,  it  is  necessary  to  give  the  pilot  all  necessary 
information  (such  as  airspeed,  altitude,  heading  and  location  of  enemy  aircraft).  This 
need  led  to  the  development  of  a  Head  Up  Display  (HUD)  which  optically  combines 
critical  information  with  an  unobstructed  view  through  the  forward  viewscreen 
[Baumann  1995]. 

In  1993,  Webb  [1993]  considered  the  appHcations  of  VR  to  Army  Training.  Some  of  the 
areas  identified  as  possibly  benefiting  from  VR  technologies  were  Hve  firing, 
parachuting,  artillery,  infantry,  and  transport. 

The  following  is  a  list  of  mUitary  appHcations  using  VR: 

•  simulating  real-world  combat  scenarios  [Alexander  1993]; 

•  mission  planning  and  rehearsal  [MosheU  et  al  1995]; 

•  training  [Durlach  et  al  1992]; 

•  training  nuclear  submarine  crews  [Alexander  1993]; 

•  training  aircraft  pilots  [Alexander  1993]; 

•  parachute  training  [Pettersen  1993]; 

•  Stinger  missile  training  [KeUy  1994]; 
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•  VETT  (Virtual  Environments  Technology  for  Training)  -  US  Navy  project  [Durlach 
etall992]; 

•  Officer-of-the-watch  training  (MARS  VRS)  [TTCP/ SGU 1996]; 

•  Hubble  Space  Telescope  Repair  Training  System  [TTCP/SGU 1996]; 

•  tele-operations  [Moshell  and  Ehinn-Roberts  1994,  KeUy  1994];  and 

•  aircraft  marshalling  (at  Standford  Research  Institute  SRI)  [Webb  1993] . 


8.  Current  Research 

The  recjuirements  of  VR  are  large,  and  there  are  many  broad  areas  in  which  research  is 
being  carried  out.  A  useful  taxonomy  of  research  area  was  given  by  Boman  [1995]  and 
this  is  listed  below; 

•  displays  for  presenting  information  to  the  user's  visual,  auditory  and  haptic  senses; 

•  sensors  and  other  technologies  for  transferring  information  from  user  to  computer; 

•  software; 

•  human  factors;  and 

•  applications. 

In  this  report,  only  the  research  being  carried  out  in  sensors  and  displays,  and  software 
wiU  be  presented. 


8.1  Sensors  and  Displays 

In  the  area  of  displays  and  sensors,  the  following  is  a  short  list  of  some  of  the  current 

activity: 

•  iimovative  optics  and  imaging  techniques  for  improving  HMDs  [Moshell  1995]; 

•  improving  haptic  interfaces  to  improve  the  sensation  of  feeling  and  touching 
through  tactile  and  force  feedback  sensations  [Moshell  and  Dunn-Roberts  1994, 
Boman  1995]; 

•  developing  improved  tracking  systems  for  tracking  the  user's  head  and  hands 
position  and  orientation  more  accurately  and  faster  as  weU  as  being  able  to  track 
other  objects  in  the  virtual  world  [Moshell  and  Dunn-Roberts  1994]; 

•  development  of  a  bodysuit  which  tracks  the  bend  in  the  various  joints  on  the 
htiman  body  [Kelly  1994]; 

•  development  of  a  visual  display  system  which  projects  an  image  directly  onto  the 
retina  using  scanning  laser  beams  [Moshell  and  Dunn-Roberts  1994,  Boman  1995]; 

•  improving  image  generators  to  reduce  the  latency  problem  (i.e.  the  delay  between  a 
change  to  the  visual  database  or  viewing  parameters  to  the  change  in  the  display) 
[Moshell  and  Dunn-Roberts  1994]; 

•  enhancement  of  VR  realism  by  considering  stereoscopic  visuals,  physical  properties 
of  objects  and  their  interaction  [KeUy  1994]; 
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•  improving  tiie  illusion  of  immersion  by  attempting  to  satisfy  all  human  senses 
[Barfield  et  al  1995,  Kelly  1994];  and 

•  improving  3D  acoustics  displays  [Boman  1995,  Encamacao  and  Gobel  1994,  Kelly 
1994].  Here,  the  requirement  is  for  a  complete  3D  sound  environment  where  soxmd 
is  spatially  locaHsed,  can  be  attached  to  virtual  objects  and  can  maintain  their 
location  as  a  participant  move  through  the  environment  [Boman  1995]. 


8.2  Software 

According  to  MosheU  and  Dxmn-Roberts  [1994],  the  main  aim  of  the  software  of  a  VR 
systems  is  to  maintain  a  real-time  model  of  the  virtual  world  and  to  be  able  to  maintain 
the  interactions  between  the  virtual  objects.  Besides  this  high  speed  graphics, 
rendering  the  images  is  also  crucial  [Boman  1995,  Pratt  et  al  1995].  In  the  past,  a  lot  of 
research  has  been  carried  out  in  making  computers  ran  better  in  real-time;  this  was  due 
to  the  development  of  the  flight  simulator. 

Different  generic  approaches  to  the  construction  of  the  virtual  environment  have  been 
proposed  by  several  researchers.  One  approach  separates  the  maintenance  of  the 
simulation  environment  from  the  rendering  of  the  graphical  database  [Boman  1995]. 
Appino  et  al  [1992]  presented  a  different  approach  for  the  creation  of  an  interactive, 
real-time,  3D  simidation.  They  organised  the  individual  elements  into  three  main  areas 
of  device  servers,  appHcation  knowledge  and  dialogue  manager.  These  three  processes 
are  modelled  as  a  collection  of  independent  processes  which  communicate  by  message 
passing  [Appino  et  al  1992].  Another  slightly  similar  approach  separates  the 
construction  into  two  layers  or  areas;  one  layer  supports  input  devices, 
communications  and  knowledge  about  the  geometrical  shape  while  the  another  layer 
organises  the  virtual  environment  along  the  principles  of  the  real  world  [Boman  1995]. 

The  following  is  a  short  list  of  software  research  being  carried  out  which  will  have  a 
direct  bearing  of  VR: 

•  3D  imaging  [Kelly  1994]; 

•  sound  features  (i.e.  which  sound  should  be  produced  and  is  it  necessary  to  have  an 
extensive  sound  library)  and  the  resulting  soxmd  of  various  objects  (say  a  glass  and 
a  table)  interacting; 

•  distributed  interactive  simulation  (DIS)  protocol  for  communicating  between 
various  computers  (which  includes  the  dead  reckoning  algorithms  which  allows  a 
simxdator  to  make  a  local  approximation  of  the  position  and  velocities  of  all  other 
simxilated  entities)  [Moshell  and  Dunn-Roberts  1994,  Stytz  et  al  1995]; 

•  interoperabihty  which  investigates  mechanisms  for  communicating  across 
computers  through  the  DIS  protocol  [Moshell  et  al  1995,  Stytz  et  al  1995]; 

•  dynarmc  terrain  (i.e.  algorithms  to  support  an  outdoor  environment  which  changes 
according  to  events  such  as  bomb  blasts)  which  include  soil  and  hydrology  models 
[MosheU  et  al  1995,  MosheU  and  Dxmn-Roberts  1994]; 

•  intxtitive  interfaces  [Boman  1995,  KeUy  1994]; 
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•  physical  simulation  (i.e.  animation  of  humans  and  animals)  [Moshell  and  Dunn- 
Roberts  1994,  Kelly  1994]; 

•  automated  and  semi-automated  forces  [Kelly  1994]; 

•  construction  of  3D  visual  terrain  database  using  sateUite  imagery  and  image 
sequences  [Encamacao  and  Gobel  1994]; 

•  computer  modelling,  animation  and  rendering  techniques  with  focus  on  rendering 
and  control  of  virtual  humans  (JACK); 

•  environmental  and  terrain  modeling  [PoHs  et  al  1995], 

•  mission  rehearsal  [Moshell  et  al  1995];  and 

•  rapid  prototyping  [Pohsetal  1995]. 


9.  Conclusions 


The  aim  of  this  report  was  to  familiarise  the  reader  with  the  key  areas  of  Virtual 
Reality  (VR).  VR  is  an  encompassment  of  an  environment  (visual,  audio,  haptic),  plus 
the  humans  within  the  environment,  plus  the  interfaces  between  the  two.  Therefore,  a 
description  of  VR  needs  a  description  of  both  human  senses,  and  computer  hardware 
and  software  technologies.  This  report  also  described  several  innovative  systems, 
appHcations  and  the  focus  of  current  research. 

Satisfying  the  human  senses  is  a  large  part  of  VR.  Many  developers  of  VR  systems 
indicate  that  the  major  stumbling  block  at  the  moment  is  the  inadequacy  of  the  haptic 
or  feeling  devices.  These  devices  are  supposed  to  give  feedback  to  the  user  from  the 
objects  inhabiting  the  world;  this  feedback  needs  to  be  both  tactile  (i.e.  texture  and  feel 
of  an  object)  as  well  as  the  force  feedback  (i.e.  mass  of  an  object). 

The  area  of  VR  is  very  broad,  as  it  encompasses  many  aspects.  As  such,  the 
developments  of  VR  has  an  impact  in  many  areas.  Obviously,  the  development  of  new 
hardware  and  software  technologies  has  an  impact  in  many  other  areas  such  as  flight 
simulation,  software  development  and  human  factors.  VR  enforces  that  being  in  an 
immersive  environment  and  therefore  being  able  to  interact  with  the  environment,  will 
lead  to  new  insights.  This  is  definitely  the  case  where  VR  is  being  used  to  visualise 
numerical  data.  By  seeing  the  virtual  world  from  within,  new  perspectives  and 
therefore,  new  discoveries,  can  be  made.  In  addition,  the  reconfigurable  nature  of  VR 
systems  is  more  cost-effective,  than  say,  building  another  simulator  or  building  a 
mock-up  of  a  virtual  world  [TTCP/SGU  1996].  As  weU,  the  testing  of  new  equipment 
prototypes  is  fairly  straightforward  in  the  virtual  world  as  virtual  mock-ups  are  easier 
to  modify  than  real  hardware. 

Of  the  existing  appHcation  areas,  the  possibility  of  using  VR  technologies  for  the 
construction  of  visual  databases  for  both  miHtary  and  civilian  appHcations  is 
intriguing.  The  construction  of  visual  databases  in  currently  very  time-consximing.  The 
two  systems,  known  as  SmartScene  and  PolyShop,  propose  interesting  solutions  to  this 
problem.  They  both  use  inunersive  technology  to  construct  a  virtual  world. 
SmartScene  puts  the  user  into  an  initially  empty  virtual  world;  from  here,  the  user  is 
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then  able  to  construct  the  virtual  world  by  selecting  objects  from  a  object  palette.  These 
objects  have  some  "inteUigence"  which  prevents,  for  example,  placing  a  tree  onto  a 
road.  PolyShop  attempts  to  exploit  the  Distributed  Interactive  Simulation  (DIS) 
communication  protocol  in  order  to  have  several  modellers  interacting  and 
constructing  the  same  virtual  world.  These  modellers  also  use  an  object  palette  and 
hvo  handed  gloves  by  which  to  construct  a  virtual  world.  The  potential  time  saving  of 
these  systems  is  enormous. 

One  of  the  problems  identified  in  the  VR  area  [Kalawsky  1993,  Moshell  and  Dunn- 
Roberts  1994,  Alexander  1993]  is  that  there  exists  no  universally  accepted  hardware 
and  software  standards.  This  has  the  affect  of  negatively  affecting  transportability  and 
compatibility  [Alexander  1993].  If  in  the  future,  virtual  worlds  are  to  interact  across 
networks,  for  instance,  using  the  DIS  communication  protocol,  then  standards  and 
interoperability  aspects  need  to  be  sorted  out. 
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